Skip to content

Distributed container hosting: verified 3-node production launch, security, metering and UX #394

Open

Description

@ralyodio

Goal

Production-grade distributed container hosting on the existing c0mpute apps implementation, not a redundant new stack. Users can deploy a Dockerfile or digest-pinned OCI image once, have independent P2P operators hold replicas, and serve a custom domain through network gateways.

Existing implementation (verify, don't rebuild)

The repository already implements:

  • c0mpute apps deploy ./api --name api --port 8080 --state /data, status/logs/secrets/destroy, apps mcp tools, git push c0mpute://....
  • Owner-signed app records with image source, resource limits, domains, host allowlists, version sequences.
  • Default three-replica rendezvous placement, sticky primary promotion, encrypted incremental snapshots, per-holder sealed app secrets.
  • Loopback-only container ports and gateway reverse-proxy RPC, limited-size buffered HTTP, DNS TXT custom-domain ownership checks.
  • Opt-in container workers (--roles apps) and per-node limits.

Docs: docs/apps.md, DIP-0031.

Launch acceptance / outstanding work

  • Secure by default: require protected egress for untrusted containers; PR fix(apps): require network egress protection by default for hosted containers #393 makes existing guard mandatory by default. Verify no IPv6 bypass, private-network SSRF, host or metadata access from inside Docker; don't rely on IPv4 iptables alone.
  • Multi-node staging: three independent Docker hosts with operator IDs, two distinct networks; verified signed app placement to 3 holders and one active primary.
  • End-to-end deployment: build a sample HTTP API from Dockerfile, publish image/chunks, deploy signed app, resolve /a/<app_id>/, verify custom domain + TXT, securely set/redeploy secrets.
  • Failure injection: kill primary, partition hosts, lose one standby, recover state, verify exactly which requests were dropped and bounded data loss; test short and sustained partitions / double-primary conflicts.
  • Isolation: resource exhaustion, admission denial, unsafe file/secret access, Docker host filesystem and socket exposure, hostile images; per-tenant disk/memory/PID/CPU and egress tests.
  • Trust limitations: owners must opt into which hosts see their secrets; encrypted snapshots protect rest, not plaintext memory on operator hosts. Enforce signed host and owner allowlists end-to-end.
  • Product surface: make CLI/API/MCP and the PWA reflect verified live capability. Add provisioning/deploy UX, owned apps dashboard, logs, status, replica health, billing events, and container plans. Don't infer health from gossip alone.
  • Payments and economics: define per-hour usage metering, signed resource receipts, pricing, settlement/dispute handling and abuse controls, while remaining competitively priced.
  • Streaming networking: add WebSockets/SSE, larger uploads, raw TCP only after security design; current gateway limits 700 KiB request, 5 MiB response and 15-second upstream timeout.
  • Release: signed cross-platform artifacts and verified dev2 deployment, plus independent security review for untrusted workloads. Urgent: dev2 deployments fail — GitHub runner SSH port 22 timeout #391 tracks SSH timeouts to dev2.

Definition of done

Independent operators can serve the same app after its owner goes offline. The app survives a single operator failure and its bounded state loss is measured, not guessed. The gateway routes to the verified primary without leaking secrets, and tenants cannot reach the underlying host or other tenants. A real new user can deploy, inspect, pay, update, and remove a service from supported CLI/API/MCP/PWA surfaces.

Not complete until tested with multiple real independent hosts. Docker locally running a container is not sufficient. A multihop VPN is separately tracked in Scallion #386/#389.

Activity

  1. ralyodio commented on Nov 17, 2026

    @ralyodio
    CollaboratorAuthor

    First hardening PR: PR #393 changes existing c0mpute apps hosting to require a guarded Docker network by default, and verifies effective network driver/name/bridge/ICC/label and IPv6-disabled status so the IPv4 iptables rule set cannot accidentally be used as evidence of protection on a mismatched or IPv6-enabled bridge. Existing explicit unsafe opt-outs remain possible but require operator intent. Security workflows pass; Rust tests still running. Production independent 3-host end-to-end deployment/failover and billing remain OPEN.

  2. ralyodio commented on Nov 17, 2026

    @ralyodio
    CollaboratorAuthor

    Distributed container hosting verification milestone: PR #396 adds a dependency-free, read-only scripts/verify-distributed-app.py that queries at least two independent gateway URLs for the same owner-signed record, 3 unique holders, healthy elected primary/epoch, gateway consistency and HTTP app health, with optional post-manual-failure validation (--previous-primary / --previous-epoch). Its dedicated Python compilation/unit-test GitHub Actions workflow has passed; full workspace Rust CI is still running. The script does not autonomously deploy/kill hosts or verify operator independence, disk snapshot contents or host firewall; real three-host testing remains OPEN.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions