Skip to content

Urgent: dev2 deployments fail — GitHub runner SSH port 22 timeout #391

Open

Description

@ralyodio

Production deployment blocker — SSH unreachable from GitHub Actions

The Deploy to dev2 workflow is failing on multiple consecutive merges.

Evidence:

Required operator diagnosis (avoid posting secrets/IPs)

On dev2, over a known working admin console/session:

sudo systemctl status ssh || sudo systemctl status sshd
sudo ss -lntp | grep ':22'
sudo ufw status verbose
sudo journalctl -u ssh -n 100 --no-pager
sudo fail2ban-client status sshd 2>/dev/null || true

From an authorized external system, check the hostname configured as the DEV2_HOST secret resolves to the intended public or reachable overlay address and TCP/22 is reachable. Also inspect upstream firewall rules, ACLs, provider networking, SSH listen port, access restrictions and recent changes. A bad key normally results in Permission denied, not a TCP connection timeout.

Do not blindly disable the firewall, change host-key verification, paste SSH secrets, or expose SSH publicly to all GitHub runner IPs. Prefer a scoped reachable endpoint, a tightly controlled overlay connection (Tailscale/WireGuard with GitHub OIDC where provisioned), or an isolated self-hosted deployment runner with appropriate access.

Acceptance

  • GitHub Actions deploy can establish an authenticated, host-key verified SSH connection to dev2.
  • The remote deploy-app.sh completes on the exact expected master SHA.
  • The workflow verifies https://c0mpute.com/ and reports success.
  • Deployed build/version is verified; operator can roll back.
  • Failure mode does not cause repeated ~130-second timeouts or claim deployment success.

Related: #386, #387, #388, #390. The signed v0.3.77 GitHub release is a separate pipeline; a published GitHub release does not mean dev2 deployment succeeded.

Activity

  1. ralyodio commented on Nov 15, 2026

    @ralyodio
    CollaboratorAuthor

    Additional company-repo finding: profullstack/threatcrush ships .github/workflows/unban-dev2.yml, which explicitly documents auto-defence IP bans on dev2 causing SSH timeouts. It describes a GitHub runner SSH workflow as break-glass, but that workflow itself relies on TCP/22 reachability and may also fail if upstream port 22 is blocked. Check the threatcrushd blocklist/remediation rules and provider firewall from the dev2 console before making changes; do not indiscriminately disable protection or whitelist all runner IPs. The two c0mpute deployment attempts timing out are consistent with a firewall/ACL/drop, but the logs alone cannot establish whether ThreatCrush, fail2ban, cloud security groups, host downtime, or another cause is responsible.

  2. ralyodio commented on Nov 17, 2026

    @ralyodio
    CollaboratorAuthor

    Live-site verification note (2026-11-16): c0mpute.com/apps public page is readable, but external attempts to open https://c0mpute.com/status returned HTTP 502 and https://c0mpute.com/apps/v1/apps could not be reached using the public fetcher. This does not conclusively prove a user-facing outage (fetcher/CDN restrictions are possible), but it does not validate the live gateway either. Combined with earlier GitHub Actions SSH TCP/22 timeouts, a trusted operator should check dev2 connectivity, edge proxy health and the actual app/gateway process before declaring distributed hosting launched. Do not change firewall/SSH rules blindly.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions