Skip to content

fix(inference): widened offers get bids; ETAs survive restarts; bridge benches failing workers (v0.3.73) - #371

Merged
ralyodio merged 3 commits into
masterfrom
fix/inference-accept-tag-live
Nov 15, 2026
Merged

ralyodio merged 3 commits into
masterfrom
fix/inference-accept-tag-live

Conversation

@ralyodio

@ralyodio ralyodio commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Follow-up to #370, from watching v0.3.71 live on the dev2 bridge.

  1. Every widened plain-text offer got 0 bids, and the bridge's exact-model retry caught each one. Cause: dispatch::live_tags rebuilds the inference tags from the runtime's model list on every offer and re-added only the role and model tags, never infernet:inference:accept. It now adds the accept tag whenever models are served; test extended.
  2. Every update reset inference ETAs to the 60 s placeholder, because inference::worker_load() skipped runner::with_history. Job times now persist like every other workload's.
  3. The jobs bridge never benched failing workers. It's the only gateway that didn't use the penalty box. sergey-234 (…mgsu3m6q) won 20 inference jobs in 24 h and failed 15: the bridge can't connect to it directly, and it reports Failed about 45 s after accepting. Each failure cost the caller a re-award of about 50 s. AuctionOutcome.failed_workers now lists workers re-awarded past. The bridge strikes them in a persistent penalty box (jobs-bridge-penalties.json: 30 min, doubling to 6 h), keeps benched workers as fallback-only bidders, and clears a worker that delivers.

🤖 Generated with Claude Code

@github-actions

Copy link
Copy Markdown

ThreatCrush Security Scan

14 finding(s)

HIGH/CRITICAL: 2 | MEDIUM: 8 | LOW: 4

Severity Rule Location
HIGH js-ssrf-outbound-request apps/web/public/sw.js:24
HIGH sh-remote-script-execution scripts/install.sh:606
MEDIUM js-unescaped-html-sink apps/web/src/app/blog/[slug]/page.tsx:53
MEDIUM js-unescaped-html-sink apps/web/src/app/layout.tsx:109
MEDIUM sql-template-interpolation apps/web/src/lib/chat/convo-store.ts:116
MEDIUM sql-template-interpolation apps/web/src/lib/chat/convo-store.ts:117
MEDIUM sql-template-interpolation apps/web/src/lib/chat/push.ts:476
MEDIUM js-dynamic-code-execution apps/web/src/lib/router/verify.ts:91
MEDIUM sh-remote-script-execution scripts/dev-setup.sh:25
MEDIUM sh-eval-expansion scripts/dev-setup.sh:35
LOW py-ssrf-outbound-request examples/business-strategist-agent/chat.py:37
LOW py-ssrf-outbound-request examples/business-strategist-agent/train.py:54
LOW secret-generic-credential node/crates/c0mpute-core/src/webhook.rs:1171
LOW py-path-traversal node/crates/c0mpute-core/tests/fixtures/fake-infernet-text:29

Snippets are redacted; ThreatCrush never prints matched credential material.

@ralyodio ralyodio changed the title fix(inference): widened offers get bids; ETAs survive restarts (v0.3.72) fix(inference): widened offers get bids; ETAs survive restarts; bridge benches failing workers (v0.3.72) Nov 15, 2026
ralyodio and others added 3 commits November 15, 2026 00:58
Live on 0.3.71 every widened plain-text offer got 0 bids and the bridge fell
back to the exact model each time. dispatch::live_tags rebuilds the inference
tags per offer from the runtime's model list (the boot snapshot predates it)
and only re-added the role and model tags, never infernet:inference:accept.
It now adds the accept tag whenever models are served.

inference::worker_load() also skipped runner::with_history, so every update
reset each node's ETA to the 60 s placeholder and the bench score alone picked
winners (a node that cannot fetch e2e specs then won and failed, costing each
job a ~50 s re-award). Job times now persist like every other workload's.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The jobs bridge, which runs every infernet.inference auction, never used
the penalty box (only the speech gateway and text training did), so a node
that cannot take jobs kept winning: sergey-234 won 20 and failed 15 in 24 h,
each failure costing the caller a ~50 s re-award.

AuctionOutcome now lists failed_workers (re-awarded past for a failed or
missing receipt). The bridge strikes them in a persistent PenaltyBox
(jobs-bridge-penalties.json: 30 min, doubling to 6 h), passes benched
workers as fallback-only bidders (they still win when nobody else bids),
and clears a worker that delivers.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
v0.3.72 went out with #372 while this PR was in CI.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sh1pt-actions-fleet
sh1pt-actions-fleet Bot force-pushed the fix/inference-accept-tag-live branch from ae8f700 to 6670285 Compare November 15, 2026 00:58
@ralyodio ralyodio changed the title fix(inference): widened offers get bids; ETAs survive restarts; bridge benches failing workers (v0.3.72) fix(inference): widened offers get bids; ETAs survive restarts; bridge benches failing workers (v0.3.73) Nov 15, 2026
@ralyodio
ralyodio merged commit 8053bc4 into master Nov 15, 2026
@ralyodio ralyodio mentioned this pull request Nov 15, 2026
ralyodio added a commit that referenced this pull request Nov 15, 2026
Ships #371 (inference: widened offers get bids, ETAs survive restarts, the
jobs bridge benches failing workers) and #374 (node contact email, message
any node with `c0mpute dm send <node>`, `dm read`, TUI inbox), merged after
the v0.3.73 tag (#376).

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
ralyodio added a commit that referenced this pull request Nov 15, 2026
…380)

* test(hosting_market_e2e): give convergence CI headroom (90 s -> 240 s)

preferred_hosts_hold_pass_challenges_and_get_paid failed on most master runs
on 2026-11-15 (and on #371/#379): the first three copies land within seconds,
but one host's 5 MiB fetch fails and retries ("chunks still missing"), the
second preferred copy arrives ~60 s in, and the extra copies still need
their trim grace, which ran past the 90 s wait on loaded runners. The test
still requires both preferred hosts and exactly three stable copies; only
the time allowed grows (agreement window 60 s -> 120 s too).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* ci(rust): retry failed tests once by exact name

Multi-node loopback E2Es time out now and then on loaded runners; on
2026-11-15 hosting_market_e2e, hosting_e2e, git_e2e and a private-objects
replication test failed across most PRs and on master. The Test step now
runs with --no-fail-fast, and only if it fails reruns each failed test once
by exact name; a test that fails twice still fails the job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant