Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
32 changes: 26 additions & 6 deletions docs/apps.md
Original file line number Diff line number Diff line change
Expand Up @@ -76,6 +76,23 @@ _c0mpute-app.api.example.org. TXT "app=<app_id>"
A gateway serves the host only when the owner-signed record lists the domain
**and** DNS agrees. Apps are tried before static sites.

## Operator safety requirements

**Default as of this change:** apps require a working egress guard.
On hosts where Docker/iptables cannot install and enforce that guard,
`apps` workers do not start app containers; this is intentional and may
reduce the available capacity until the operator provisions protected
workers. Existing configurations that explicitly contain
`require_egress_guard = false` remain opted out; review them before
running untrusted workloads. For an untrusted marketplace, also configure
`[apps] owners` and the signed `hosts` allowlist to constrain which
operators can access app secrets and runtime memory.

A successful unit test does not validate your host's iptables,
Docker bridge, IPv6 routing, side-channel isolation, or failover
behavior. Verify those on actual hosts before accepting untrusted
containers.

## Running apps for the network

`apps` is **opt-in** (it runs other people's containers and needs docker):
Expand All @@ -90,7 +107,7 @@ max_apps = 4 # held at once (primary or standby)
max_memory_mb = 2048 # reserved by held apps together
owners = [] # owner keys whose apps this node runs; empty = anyone's
egress_guard = true # keep apps off your LAN, the host and metadata addresses
require_egress_guard = false # true: run no apps unless that guard is in place
require_egress_guard = true # default: run no apps unless that guard is in place
max_disk_mb = 2048 # per app: state volume plus writes inside the container
```

Expand All @@ -102,11 +119,14 @@ the network, and says so in its log at start.
**Network.** The egress guard is a set of iptables rules on the apps bridge
(`c0apps0`), the same ones `container.run` uses plus one that lets the app
answer the node's own proxy. Only a root worker can add them. A non-root
worker keeps apps on docker's default bridge, where a container **can reach
your LAN, the host and cloud metadata addresses**; the node logs that once at
start. Set `require_egress_guard = true` to run no apps in that case. An app
that must reach a private address (a database on your LAN) needs
`egress_guard = false`. The guard is IPv4 only.
worker **refuses to run app containers by default** rather than placing
untrusted workloads on Docker's unguarded bridge. The operator can explicitly
set `require_egress_guard = false` to opt out, but that can give containers
access to the host, LAN and cloud metadata addresses. An app needing a private
LAN database must be run on a trusted dedicated host with explicit restrictions:
`egress_guard = false` **and** `require_egress_guard = false`; do not expose
that host to arbitrary app owners. The current guard is IPv4 only; treat
IPv6 exposure as unverified until an IPv6 policy is implemented and tested.

**Disk.** Every minute the node asks docker how much a running app has written
inside its container and adds the size of its last state snapshot. Over
Expand Down
76 changes: 76 additions & 0 deletions node/crates/c0mpute-core/src/apps/runtime.rs
Original file line number Diff line number Diff line change
Expand Up @@ -136,6 +136,34 @@ pub fn guard_rules(bridge: &str) -> Vec<Vec<String>> {
rules
}

/// Accept only the exact Docker bridge we created. An existing network
/// with the same name can otherwise be attached to a different interface,
/// bypassing the iptables rules on c0apps0. Since our rules are IPv4-only,
/// reject any IPv6-enabled bridge instead of falsely claiming isolation.
pub fn inspect_guarded_network(value: &serde_json::Value) -> Result<()> {
use anyhow::ensure;

let network = value.as_array()
.and_then(|entries| entries.first())
.ok_or_else(|| anyhow::anyhow!("missing Docker network description"))?;
ensure!(network["Name"].as_str() == Some(APPS_NETWORK), "wrong Docker network name");
ensure!(network["Driver"].as_str() == Some("bridge"), "apps require a Docker bridge");
ensure!(network["EnableIPv6"].as_bool() == Some(false), "IPv6-enabled bridge has no verified egress firewall");
ensure!(
network["Options"]["com.docker.network.bridge.name"].as_str() == Some(APPS_BRIDGE),
"apps network uses unexpected bridge; IPv4 firewall cannot protect it"
);
ensure!(
network["Options"]["com.docker.network.bridge.enable_icc"].as_str() == Some("false"),
"apps network allows inter-container communication"
);
ensure!(
network["Labels"]["c0mpute.apps"].as_str() == Some("1"),
"apps network ownership label is missing"
);
Ok(())
}

/// Make sure the apps network exists with the egress guard on it. True when
/// app containers can be put behind the guard; false for a non-root worker
/// (it cannot change the firewall) or when docker or iptables refuse.
Expand All @@ -147,6 +175,21 @@ pub async fn ensure_guarded_network() -> bool {
tracing::warn!(err = %format!("{e:#}"), "apps: could not create the apps network");
return false;
}
// Check the effective Docker network settings (including preexisting
// networks), not just whether docker network inspect returned success.
let inspect = docker(
&["network".into(), "inspect".into(), APPS_NETWORK.into()],
None,
Duration::from_secs(15),
).await;
let verified = inspect.and_then(|bytes| {
let value: serde_json::Value = serde_json::from_slice(&bytes)?;
inspect_guarded_network(&value)
});
if let Err(e) = verified {
tracing::warn!(err = %format!("{e:#}"), "apps: refusing unverified bridge or IPv6 egress");
return false;
}
crate::container_run::ensure_rules(guard_rules(APPS_BRIDGE)).await
}

Expand Down Expand Up @@ -709,6 +752,39 @@ mod tests {
assert_eq!(a.iter().filter(|x| *x == "--network").count(), 1);
}

#[test]
fn a_reused_or_ipv6_enabled_docker_bridge_is_not_considered_guarded() {
let safe = serde_json::json!([{
"Name": "c0mpute-apps",
"Driver": "bridge",
"EnableIPv6": false,
"Options": {
"com.docker.network.bridge.name": "c0apps0",
"com.docker.network.bridge.enable_icc": "false"
},
"Labels": {"c0mpute.apps": "1"}
}]);
assert!(inspect_guarded_network(&safe).is_ok());

for (path, unsafe_value) in [
(vec!["0", "EnableIPv6"], serde_json::json!(true)),
(vec!["0", "Driver"], serde_json::json!("macvlan")),
(vec!["0", "Name"], serde_json::json!("bridge")),
(vec!["0", "Options", "com.docker.network.bridge.name"], serde_json::json!("br0")),
(vec!["0", "Options", "com.docker.network.bridge.enable_icc"], serde_json::json!("true")),
(vec!["0", "Labels", "c0mpute.apps"], serde_json::json!("0")),
] {
let mut network = safe.clone();
let mut current = &mut network[0];
for segment in &path[1..path.len() - 1] {
current = &mut current[*segment];
}
current[path[path.len() - 1]] = unsafe_value;
assert!(inspect_guarded_network(&network).is_err(), "{path:?}");
}
assert!(inspect_guarded_network(&serde_json::json!([])).is_err());
}

#[test]
fn the_guard_decides_the_network_and_require_refuses_without_it() {
assert_eq!(network_for(true, false), Ok(Some(APPS_NETWORK)));
Expand Down
26 changes: 24 additions & 2 deletions node/crates/c0mpute-core/src/config.rs
Original file line number Diff line number Diff line change
Expand Up @@ -221,7 +221,9 @@ pub struct AppsConfig {
/// `container.run` uses (needs root; without it apps stay on docker's
/// default bridge and the node says so at start).
pub egress_guard: bool,
/// Run no apps at all unless that guard is in place.
/// Refuse unguarded container network access by default. Operators must
/// explicitly accept LAN/host/metadata exposure to opt out, and should
/// then restrict owners to trusted identities.
pub require_egress_guard: bool,
/// Most disk one app may use, MiB: its state volume plus what it writes
/// inside its container. An app over it is stopped.
Expand All @@ -235,7 +237,7 @@ impl Default for AppsConfig {
max_memory_mb: 2048,
owners: Vec::new(),
egress_guard: true,
require_egress_guard: false,
require_egress_guard: true,
max_disk_mb: crate::apps::DEFAULT_MAX_DISK_MB,
}
}
Expand Down Expand Up @@ -1165,6 +1167,26 @@ pub fn write_atomic(path: &Path, bytes: &[u8]) -> std::io::Result<()> {
mod tests {
use super::*;

#[test]
fn apps_fail_closed_without_an_egress_guard_by_default() {
let defaults = AppsConfig::default();
assert!(defaults.egress_guard);
assert!(defaults.require_egress_guard);

// Existing config files that omit the new field take the safer default.
let parsed: AppsConfig = toml::from_str("max_apps = 4").unwrap();
assert!(parsed.require_egress_guard);
assert!(parsed.egress_guard);

// An operator can still make a deliberate, documented opt-out.
let explicit: AppsConfig = toml::from_str(
"egress_guard = false\nrequire_egress_guard = false",
).unwrap();
assert!(!explicit.require_egress_guard);
assert!(!explicit.egress_guard);
}


#[test]
fn save_replaces_the_config_whole_and_keeps_the_old_one_when_it_cannot() {
let dir = tempfile::tempdir().unwrap();
Expand Down
Loading