Skip to content

setup_docker: make LXC resource limits visible to nested containers - #9

Open
andrebrait wants to merge 1 commit into
community-scripts:mainfrom
andrebrait:fix/docker-lxc-resource-visibility
Open

setup_docker: make LXC resource limits visible to nested containers#9
andrebrait wants to merge 1 commit into
community-scripts:mainfrom
andrebrait:fix/docker-lxc-resource-visibility

Conversation

@andrebrait

Copy link
Copy Markdown

✍️ Description

Inside an LXC container, Docker gives every container it starts a fresh procfs in its own mount namespace. The lxcfs bind mounts that LXC placed on /proc/meminfo and friends live in the CT's namespace and never reach it, so every Docker container reads the physical host's totals instead of the CT's limits:

CT sees:        MemTotal:  8388608 kB     # the CT's 8 GiB
container sees: MemTotal: 65648168 kB     # the whole 62 GiB hypervisor
cgroup limit:   max

Enforcement was never affected — the CT cgroup still caps every descendant. Visibility was, and anything that self-sizes from those numbers (JVM heap ergonomics, Node, Go, OpenMP thread pools) sizes against the wrong value and gets OOM-killed instead of throttling.

This registers a small runc wrapper as Docker's default-runtime. On each container create it re-binds the lxcfs-backed files into the container and, when the container sets no limit of its own, applies the CT's memory limit to the container cgroup.

Both halves are needed, because runtimes read the limit two different ways:

  • glibc's sysconf(_SC_PHYS_PAGES), procps free and friends read /proc/meminfo → fixed by the binds.
  • musl and Node's uv_get_constrained_memory() consult the cgroup → fixed by the limit.

CPU needs no handling: cpuset is inherited down the cgroup tree, so nested containers already observe the CT's core count. The one exception is /sys/devices/system/cpu/online, which glibc's sysconf(_SC_NPROCESSORS_ONLN) reads, so it is bound alongside the /proc files. Without it, a container on a 2-core CT reports getconf _NPROCESSORS_ONLN = 6 on a 6-core host while nproc correctly says 2 (coreutils uses sched_getaffinity).

Toggle

DOCKER_LXCFS_VISIBILITY, default true, documented in the setup_docker() header alongside the existing USE_DOCKER_REPO / DOCKER_LOG_DRIVER / DOCKER_SKIP_UPDATES. lxc-attach inherits the environment, so users opt out with:

DOCKER_LXCFS_VISIBILITY=false bash -c "$(curl -fsSL .../ct/docker.sh)"

Safety

  • No-op outside an LXC container and wherever lxcfs is not mounted (guard: grep -qs 'fuse\.lxcfs' /proc/mounts), so the five non-Docker installers that call setup_docker are unaffected.
  • Fails open. No runc, no jq, malformed JSON, or empty output all fall through to the real runc unmodified — worst case is current behaviour, never a dead daemon.
  • runc resolved at runtime, since distro docker.io ships /usr/sbin/runc while containerd.io ships /usr/bin/runc.
  • Never overrides an explicit --memory / mem_limit.
  • docker run --runtime=runc bypasses the wrapper for a single container.
  • Runs before systemctl enable -q --now docker, so the daemon starts with the runtime already registered — no restart needed.
  • The jq merge preserves any existing daemon.json keys (registry-mirrors, insecure-registries, log-driver).

Known limitation, documented in the function header

The limit is written at container creation. After changing a CT's memory with pct set, existing containers keep the old value until recreated, or re-synced with docker update --memory=<new> --memory-swap=<new> <container>. New containers pick up the new value immediately, and CPU changes propagate live with no action.

The alternative — binds only, no cgroup limit — tracks resizes live (lxcfs walks up to the nearest ancestor with a limit) but leaves Node and musl reading the host. I judged that the worse default; happy to flip it if maintainers disagree.

🔗 Related Issue

Fixes community-scripts/ProxmoxVE#16756

✅ Prerequisites (X in brackets)

  • Self-review completed – Code follows project standards.
  • Tested thoroughly – Changes work as expected (Proxmox VE and/or Incus, as applicable).
  • No security risks – No hardcoded secrets, unnecessary privilege escalations, or permission issues.
  • API.txt regenerated – If a function signature or behavior changed, API.txt was updated to match.

🤖 AI Assistance (X in brackets)

  • No AI used – Code was written without AI assistance.
  • AI was used – The output has been reviewed and corrected, and I take responsibility for it matching this repo's conventions.

Model: Claude Opus 5 (1M context), reasoning effort high, via Claude Code.


🛠️ Type of Change (X in brackets)

  • 🐞 Bug fix – Resolves an issue without breaking functionality.
  • New feature – Adds new, non-breaking functionality.
  • 💥 Breaking change – Alters existing functionality in a way that may require updates.
  • 🆕 New script – A fully functional and tested script or script set.
  • 🌍 Website update – Changes to script metadata (PocketBase/website data).
  • 🔧 Refactoring / Code Cleanup – Improves readability or maintainability without changing functionality.
  • 📝 Documentation update – Changes to README, AppName.md, CONTRIBUTING.md, or other docs.

Test evidence

Run against a real Proxmox VE 9.2.10 host (PVE kernel 7.0.14-11-pve, lxcfs 7.0.0-pve1) in an unprivileged Debian 13 CT with features: nesting=1, 8192 MB / 6 cores, Docker 29.7.2, on a 62 GiB host. The CT was reset to a clean pre-patch state, then the patched _docker_setup_lxcfs_visibility was extracted from lib/runtime.func and executed:

RED: default=runc
MemTotal:       65648168 kB
max

[info] Configuring LXC resource visibility for Docker
[ok] Configured LXC resource visibility for Docker

GREEN: default=lxcfs
MemTotal:        8388608 kB
8589934592

Additional cases, same CT:

[B --memory=1g respected]  MemTotal:  1048576 kB   memory.max: 1073741824
[C --runtime=runc bypass]  MemTotal: 65648168 kB   memory.max: max
[D cpu/online bound]       1 mount   (8 lxcfs binds total)
[E second run idempotent]  {"dr":"lxcfs","rt":["lxcfs"],"mirrors":1}
[F toggle off]             returned 0, no action taken
[G survives daemon restart] MemTotal: 8388608 kB   memory.max: 8589934592

Note [B]: lxcfs resolves per reading process's cgroup, so a container with its own --memory reports its limit through the same bind — no per-container wiring needed.

CI gates run locally:

shellcheck --shell=bash --severity=error --exclude=SC1091 --format=gcc lib/runtime.func   -> rc=0
shellcheck --severity=warning, added range                                               -> clean
verify-split-files API check: lib/tools.func vs lib/API.txt                               -> matches (151 functions)

Verification one-liner for reviewers

echo "CT sees:        $(head -1 /proc/meminfo)"
echo "container sees: $(docker run --rm alpine head -1 /proc/meminfo)"
echo "cgroup limit:   $(docker run --rm alpine cat /sys/fs/cgroup/memory.max)"

Two commands that look fine while this is broken, worth avoiding: docker info reports the correct MemTotal throughout, because the daemon is a plain CT process reading lxcfs; and docker run --rm alpine free -m prints the host figure even once fixed, because busybox free uses sysinfo(2), which lxcfs cannot intercept.

@github-actions

Copy link
Copy Markdown

Try this branch

The engine and the scripts resolve independently, so a production script can
be run against the engine from this PR by setting one variable:

COMMUNITY_SCRIPTS_CORE_URL=https://raw.githubusercontent.com/andrebrait/core/fix/docker-lxc-resource-visibility \
bash -c "$(curl -fsSL https://raw.githubusercontent.com/community-scripts/ProxmoxVED/main/ct/debian.sh)"

Swap ct/debian.sh for whatever exercises the change.

Run a script from a fork as well
curl -fsSL https://raw.githubusercontent.com/andrebrait/core/fix/docker-lxc-resource-visibility/tools/run.sh |
  bash -s -- https://raw.githubusercontent.com/YOU/ProxmoxVED/your-branch ct/debian.sh \
             https://raw.githubusercontent.com/andrebrait/core/fix/docker-lxc-resource-visibility

Note that run.sh is reached through a pipe, so the script it starts inherits
an exhausted stdin. Whiptail is fine — it opens /dev/tty — but a plain read
would see EOF. The single-variable form above does not have that problem.

Useful flags while testing

dev_mode=net logs every engine fetch with status and duration, which is the
quickest way to confirm the branch is really being used. dev_mode=keep stops a
failed build from deleting the container along with the evidence.

Inside an LXC container, Docker gives every container it starts a fresh
procfs in its own mount namespace. The lxcfs bind mounts that LXC placed
on /proc/meminfo and friends live in the CT's namespace and never reach
it, so each Docker container reads the physical host's totals instead of
the CT's limits:

  CT sees:        MemTotal:  8388608 kB
  container sees: MemTotal: 65648168 kB

Enforcement was never affected, since the CT cgroup still caps every
descendant. Visibility was, and anything that self-sizes from those
numbers - JVM heap ergonomics, Node, Go, OpenMP thread pools - sizes
against the wrong value and gets OOM-killed instead of throttling.

Register a small runc wrapper as Docker's default runtime. On each
container create it re-binds the lxcfs-backed files into the container
and, when the container sets no limit of its own, applies the CT's
memory limit to the container cgroup. Both halves are needed because
runtimes read the limit two different ways: file readers use /proc,
while musl and Node's uv_get_constrained_memory() consult the cgroup.

CPU needs no handling - cpuset is inherited down the cgroup tree, so
nested containers already observe the CT's core count. The one exception
is /sys/devices/system/cpu/online, which glibc's sysconf(_SC_NPROCESSORS_ONLN)
reads, so it is bound alongside the /proc files.

The helper is a no-op outside an LXC container and wherever lxcfs is not
mounted, so the five non-Docker installers that call setup_docker are
unaffected. It fails open: no runc, no jq, or malformed JSON all fall
through to the real runc unmodified, degrading to current behaviour
rather than breaking the daemon. Opt out with DOCKER_LXCFS_VISIBILITY=false.

Regenerate lib/API.txt for the new function.
@andrebrait
andrebrait force-pushed the fix/docker-lxc-resource-visibility branch from b644b18 to c2dbd48 Compare August 25, 2026 16:43
@andrebrait

Copy link
Copy Markdown
Author

Adding a caveat found in live testing after opening this, now documented in the function header.

Monitoring containers are the one case that wants the opposite of this patch.

lxcfs resolves usage by the reading process's own cgroup. Once /proc/meminfo is bound in, a container that graphs "system memory used" reports only its own footprint. MemTotal is correct; MemFree/MemAvailable/Cached are not. Measured on a 32 GiB CT with ~290 MB in use:

CT itself:           MemFree: 33254596 kB   Cached: 164344 kB
with these binds:    MemFree: 33553052 kB   Cached:      0 kB

Two things worth noting about the scope of this:

  • Dropping the cgroup limit does not avoid it. I checked, expecting usage to walk up to the CT the way MemTotal does. It does not — only the limit walks up; usage stays scoped to the reading container's leaf cgroup either way. So this is inherent to lxcfs, not a consequence of the memory.limit half of the patch.
  • CPU is unaffected. /proc/stat is not cgroup-scoped; the cpu line inside a bound container tracks the CT's within a few ticks.

Tools that read the Docker API rather than /proc are fine, because the daemon is a plain CT process reading the CT's own lxcfs values — that covers Portainer's dashboard and Arcane's Docker info panel. Tools using gopsutil's mem.VirtualMemory() (Arcane's live system-stats websocket, netdata, glances) are affected.

Escape hatch is per container, no config change needed elsewhere:

services:
  arcane:
    runtime: runc

I don't think this should change the default. The patch fixes every workload that self-sizes from the wrong number, which is the common case; monitoring containers are a small, identifiable set with a one-line opt-out. But it is a real behaviour change for them and maintainers should weigh it — happy to gate it differently if you'd prefer, e.g. skipping containers that mount /var/run/docker.sock.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

[Docker] LXC resource limits ignored by Docker running inside jt

1 participant