Skip to content

feat(routing): free nodes beat working ones; plain-text prompts may take a smaller model (v0.3.71) - #370

Merged
ralyodio merged 1 commit into
masterfrom
feat/smart-inference-routing
Nov 15, 2026
Merged

ralyodio merged 1 commit into
masterfrom
feat/smart-inference-routing

Conversation

@ralyodio

Copy link
Copy Markdown
Collaborator

Anthony: "how come only 3 nodes have completed jobs? we have 12", "routing is not smart/dynamic enough to pick a free node vs. a working node", and "plain text prompts can be routed to smaller models if free over a working node … all surfaces".

What 6 h of bridge logs showed (dev2, 14 Nov): 325 infernet.inference jobs. Only 3 of the 12 nodes bid: bonita (…fdxBAdC8), vienna (…AmaKBdpB) and sergey-234 (…mgsu3m6q). Bonita won 300. 318 of the jobs asked for qwen2.5:7b.

Causes

  1. A working node quoted the same ETA as an idle one, because eta_seconds ignored running jobs. Bonita quoted 87 s with a job in progress, so it won until both of its slots were full.
  2. A worker only bids when it serves the exact model requested, and it never pulls models. dev2's five workers and vienna's container worker run explicit --roles lists without inference (on purpose). Outside nodes under 16 GiB only ever get qwen2.5:1.5b, so they can't match a qwen2.5:7b request.

Changes (one place for every surface: Infernet, c0mpute.com, apps, TUI and MCP all run chat through the jobs bridge)

  • dispatch::eta_seconds_busy: each running job stretches the quoted ETA by 1/total_slots, so a half-busy node quotes 1.5x.
  • inference::with_alternatives (bridge): a plain-text prompt for a standard model also accepts smaller standard models (FALLBACK_LADDER, e.g. qwen2.5:7b → qwen3:4b, qwen2.5:3b, qwen2.5:1.5b). These keep the exact model: agent tool protocols (<tool, tool_result, tool_call, function_call), JSON-output prompts, prompts over 16 KiB, and exact_model: true. 0.5b is never used as a fallback.
  • Workers bid with the best model they serve (choose_model) and put it on the bid as JobBid.model and fallback. They run the job with that model.
  • routing: a fallback bid yields while a worker serving the asked-for model is idle. Once those are all working, the fastest free fallback wins.

Compatibility

  • Older workers: widened offers require infernet:inference:accept, which only 0.3.71+ workers advertise, so an older worker never wins a job it would refuse once opened.
  • No bids: with no bid from a 0.3.71+ worker, the bridge reruns the offer for the exact model.
  • Old bids: bids without the new fields count as the asked-for model. Both fields are optional with serde defaults, as load and instance were.

Tests: new unit tests in inference.rs (model choice, plain-text classifier, widening, required tags), routing.rs (yield while a preferred-model worker is idle, win once they're all working, old bids) and dispatch.rs (busy ETA). CI runs the workspace build and tests; nothing was built on dev2.

🤖 Generated with Claude Code

…ake a smaller model (v0.3.71)

Only 3 of 12 nodes ever bid on infernet.inference, and bonita won 92% of
jobs. Two causes, both fixed here, on every surface (Infernet, c0mpute.com,
apps, TUI, MCP all run chat through the jobs bridge):

- A working node quoted the same ETA as an idle one (eta ignored running
  jobs), so the fastest box won every auction until both its slots filled.
  eta_seconds_busy stretches the quote by busy/total_slots.
- A worker only bid when it served the exact model asked for, and nodes
  under 16 GiB only have qwen2.5:1.5b. The jobs bridge now widens a
  plain-text prompt for a standard model with smaller standard models
  (inference::FALLBACK_LADDER; agent tool protocols, JSON output, prompts
  over 16 KiB and exact_model:true keep the exact model). Workers bid with
  the best model they serve and say so on the bid (JobBid.model/fallback).
  The buyer keeps the asked-for model while a worker serving it is idle;
  once those are all working, a free alternative wins.

Compatibility: widened offers require infernet:inference:accept, which only
0.3.71+ workers advertise, so an older worker never wins a job it would
refuse; with no such bid the bridge reruns the offer for the exact model.
Old bids (no model fields) count as the asked-for model.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

ThreatCrush Security Scan

14 finding(s)

HIGH/CRITICAL: 2 | MEDIUM: 8 | LOW: 4

Severity Rule Location
HIGH js-ssrf-outbound-request apps/web/public/sw.js:24
HIGH sh-remote-script-execution scripts/install.sh:606
MEDIUM js-unescaped-html-sink apps/web/src/app/blog/[slug]/page.tsx:53
MEDIUM js-unescaped-html-sink apps/web/src/app/layout.tsx:109
MEDIUM sql-template-interpolation apps/web/src/lib/chat/convo-store.ts:116
MEDIUM sql-template-interpolation apps/web/src/lib/chat/convo-store.ts:117
MEDIUM sql-template-interpolation apps/web/src/lib/chat/push.ts:476
MEDIUM js-dynamic-code-execution apps/web/src/lib/router/verify.ts:91
MEDIUM sh-remote-script-execution scripts/dev-setup.sh:25
MEDIUM sh-eval-expansion scripts/dev-setup.sh:35
LOW py-ssrf-outbound-request examples/business-strategist-agent/chat.py:37
LOW py-ssrf-outbound-request examples/business-strategist-agent/train.py:54
LOW secret-generic-credential node/crates/c0mpute-core/src/webhook.rs:1171
LOW py-path-traversal node/crates/c0mpute-core/tests/fixtures/fake-infernet-text:29

Snippets are redacted; ThreatCrush never prints matched credential material.

@ralyodio
ralyodio merged commit 2762986 into master Nov 15, 2026
19 of 31 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant