Repository navigation
feat(routing): free nodes beat working ones; plain-text prompts may take a smaller model (v0.3.71) - #370
Merged
Conversation
…ake a smaller model (v0.3.71) Only 3 of 12 nodes ever bid on infernet.inference, and bonita won 92% of jobs. Two causes, both fixed here, on every surface (Infernet, c0mpute.com, apps, TUI, MCP all run chat through the jobs bridge): - A working node quoted the same ETA as an idle one (eta ignored running jobs), so the fastest box won every auction until both its slots filled. eta_seconds_busy stretches the quote by busy/total_slots. - A worker only bid when it served the exact model asked for, and nodes under 16 GiB only have qwen2.5:1.5b. The jobs bridge now widens a plain-text prompt for a standard model with smaller standard models (inference::FALLBACK_LADDER; agent tool protocols, JSON output, prompts over 16 KiB and exact_model:true keep the exact model). Workers bid with the best model they serve and say so on the bid (JobBid.model/fallback). The buyer keeps the asked-for model while a worker serving it is idle; once those are all working, a free alternative wins. Compatibility: widened offers require infernet:inference:accept, which only 0.3.71+ workers advertise, so an older worker never wins a job it would refuse; with no such bid the bridge reruns the offer for the exact model. Old bids (no model fields) count as the asked-for model. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ThreatCrush Security Scan14 finding(s) HIGH/CRITICAL: 2 | MEDIUM: 8 | LOW: 4
Snippets are redacted; ThreatCrush never prints matched credential material. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Anthony: "how come only 3 nodes have completed jobs? we have 12", "routing is not smart/dynamic enough to pick a free node vs. a working node", and "plain text prompts can be routed to smaller models if free over a working node … all surfaces".
What 6 h of bridge logs showed (dev2, 14 Nov): 325
infernet.inferencejobs. Only 3 of the 12 nodes bid: bonita (…fdxBAdC8), vienna (…AmaKBdpB) and sergey-234 (…mgsu3m6q). Bonita won 300. 318 of the jobs asked forqwen2.5:7b.Causes
eta_secondsignored running jobs. Bonita quoted 87 s with a job in progress, so it won until both of its slots were full.--roleslists without inference (on purpose). Outside nodes under 16 GiB only ever getqwen2.5:1.5b, so they can't match aqwen2.5:7brequest.Changes (one place for every surface: Infernet, c0mpute.com, apps, TUI and MCP all run chat through the jobs bridge)
dispatch::eta_seconds_busy: each running job stretches the quoted ETA by 1/total_slots, so a half-busy node quotes 1.5x.inference::with_alternatives(bridge): a plain-text prompt for a standard model also accepts smaller standard models (FALLBACK_LADDER, e.g.qwen2.5:7b→qwen3:4b,qwen2.5:3b,qwen2.5:1.5b). These keep the exact model: agent tool protocols (<tool,tool_result,tool_call,function_call), JSON-output prompts, prompts over 16 KiB, andexact_model: true. 0.5b is never used as a fallback.choose_model) and put it on the bid asJobBid.modelandfallback. They run the job with that model.routing: a fallback bid yields while a worker serving the asked-for model is idle. Once those are all working, the fastest free fallback wins.Compatibility
infernet:inference:accept, which only 0.3.71+ workers advertise, so an older worker never wins a job it would refuse once opened.loadandinstancewere.Tests: new unit tests in
inference.rs(model choice, plain-text classifier, widening, required tags),routing.rs(yield while a preferred-model worker is idle, win once they're all working, old bids) anddispatch.rs(busy ETA). CI runs the workspace build and tests; nothing was built on dev2.🤖 Generated with Claude Code