Skip to content

Fix CI campaign counting and lifecycle races - #28

Merged
jeregrine merged 1 commit into
mainfrom
fix-ci-campaign-harness
Sep 21, 2026
Merged

jeregrine merged 1 commit into
mainfrom
fix-ci-campaign-harness

Conversation

@jeregrine

Copy link
Copy Markdown
Member

Problem

The September 16–21 nightly campaigns failed repeatedly even when Jepsen accepted the histories. The campaign wrapper depends on rg, which is absent from the runners, and suppresses the command failure before reporting zero successful histories.

There are also lifecycle synchronization races: concurrent Jepsen setup threads heal every container while neighboring threads restart them, and two ExUnit regressions treat process registration or route discovery as proof that startup repair or anti-entropy has finished.

Fix

  • Count successful histories with standard awk, preserving failures from the counter rather than treating them as zero matches. The required history count remains unchanged.
  • Restart each container before healing only that node's firewall. Teardown also operates only on its own node, avoiding cross-node setup interference without swallowing firewall errors.
  • Wait for the restarted shard to finish initialization before inspecting the deliberately interrupted snapshot's ETS rows. Replica traffic stays dropped, so the assertions still require startup repair to remove the partial image.
  • Require deleted rows to be absent and retained rows to be present in the same reconnect convergence check. Route discovery alone does not prevent replay from briefly exposing an old join before its leave.

Regression coverage exercises a runner without ripgrep, incorrect history counts, command failures, and concurrent node setup.

Supporting information

The September 21 campaign contains both the missing-rg failures and an attempt to execute iptables in a stopped container.

This addresses the confirmed harness defects and synchronization races, not every observed nightly failure. The many-cluster and adversarial convergence timeouts, distribution/mixed checker mismatches, and TCP/permanent latency violation still need investigation. No production replication behavior, checker invariant, latency limit, convergence timeout, or soak duration is changed.

@jeregrine
jeregrine merged commit 890c046 into main Sep 21, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant