Skip to content

ci: retry flaky E2E tests once; hosting_market_e2e gets CI headroom - #380

Merged
ralyodio merged 2 commits into
masterfrom
fix/hosting-market-e2e
Nov 15, 2026
Merged

ralyodio merged 2 commits into
masterfrom
fix/hosting-market-e2e

Conversation

@ralyodio

@ralyodio ralyodio commented Nov 15, 2026 •

Copy link
Copy Markdown
Collaborator

Anthony: "the github action … PR checks failed, fix it".

Multi-node loopback E2E tests time out now and then on loaded GitHub runners. Today hosting_market_e2e, hosting_e2e, git_e2e and the private-objects replication test failed across most PRs and on master, one or another each time.

  1. ci(rust): failed tests get one retry by exact name. The Test step runs cargo test --workspace --locked --no-fail-fast. Only if that fails does it rerun each failed test once (cargo test --workspace --locked -- --exact <name>). A test that fails twice still fails the job, and a failure with no test name (a build or harness error) fails immediately. Retries show as a ::warning:: annotation, so flaky tests stay visible.
  2. hosting_market_e2e gets convergence headroom: 90 s → 240 s, and the agreement window 60 s → 120 s. In the failing logs, one host's 5 MiB fetch fails and retries (chunks still missing), the second preferred copy lands about 60 s in, and the extra copies still need their trim grace. The assertions are unchanged.

🤖 Generated with Claude Code

preferred_hosts_hold_pass_challenges_and_get_paid failed on most master runs
on 2026-11-15 (and on #371/#379): the first three copies land within seconds,
but one host's 5 MiB fetch fails and retries ("chunks still missing"), the
second preferred copy arrives ~60 s in, and the extra copies still need
their trim grace, which ran past the 90 s wait on loaded runners. The test
still requires both preferred hosts and exactly three stable copies; only
the time allowed grows (agreement window 60 s -> 120 s too).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown

ThreatCrush Security Scan

14 finding(s)

HIGH/CRITICAL: 2 | MEDIUM: 8 | LOW: 4

Severity Rule Location
HIGH js-ssrf-outbound-request apps/web/public/sw.js:24
HIGH sh-remote-script-execution scripts/install.sh:611
MEDIUM js-unescaped-html-sink apps/web/src/app/blog/[slug]/page.tsx:53
MEDIUM js-unescaped-html-sink apps/web/src/app/layout.tsx:109
MEDIUM sql-template-interpolation apps/web/src/lib/chat/convo-store.ts:116
MEDIUM sql-template-interpolation apps/web/src/lib/chat/convo-store.ts:117
MEDIUM sql-template-interpolation apps/web/src/lib/chat/push.ts:476
MEDIUM js-dynamic-code-execution apps/web/src/lib/router/verify.ts:91
MEDIUM sh-remote-script-execution scripts/dev-setup.sh:25
MEDIUM sh-eval-expansion scripts/dev-setup.sh:35
LOW py-ssrf-outbound-request examples/business-strategist-agent/chat.py:37
LOW py-ssrf-outbound-request examples/business-strategist-agent/train.py:54
LOW secret-generic-credential node/crates/c0mpute-core/src/webhook.rs:1171
LOW py-path-traversal node/crates/c0mpute-core/tests/fixtures/fake-infernet-text:29

Snippets are redacted; ThreatCrush never prints matched credential material.

Multi-node loopback E2Es time out now and then on loaded runners; on
2026-11-15 hosting_market_e2e, hosting_e2e, git_e2e and a private-objects
replication test failed across most PRs and on master. The Test step now
runs with --no-fail-fast, and only if it fails reruns each failed test once
by exact name; a test that fails twice still fails the job.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@ralyodio ralyodio changed the title test(hosting_market_e2e): give convergence CI headroom (90 s → 240 s) ci: retry flaky E2E tests once; hosting_market_e2e gets CI headroom Nov 15, 2026
@ralyodio
ralyodio merged commit 8ade06a into master Nov 15, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant