feat(core): apply max_open_files as RLIMIT_NOFILE in the child

[dzerik]

Closes #176.

max_open_files was accepted by the builder, the CLI, the C ABI, Python and Go, carried through every structure, and never applied — RLIMIT_NOFILE appeared nowhere in the crate. A caller who set it believed a descriptor cap was in place while the guest ran with the host's limit.

Where the call goes

Last in confine_child, deliberately. RLIMIT_NOFILE bounds the fd number the kernel hands out, not the count of live descriptors, and every earlier step still opens files: Landlock takes an O_PATH fd per rule path plus the ruleset fd, seccomp allocates the notify listener, close_fds_above reads /proc/self/fd. Setting the cap before those turns them into EMFILE and kills the child with a message blaming Landlock. After that step the table is {0,1,2} plus kept fds, which is what a caller means by "at most N open files". It sits before the entry match so the in-process entrypoint gets it too.

Both the soft and the hard limit are lowered: setrlimit/prlimit64 are not in the seccomp blocklist, so a soft-only cap would be advisory.

Two decisions worth your explicit attention

1. The clamp takes both inherited bounds, which diverges from container convention.

Clamping to the hard limit alone made the setting widen the budget: with the common soft 1024 / hard 524288 split, --max-open-files 65536 handed the guest 64x the descriptors the same command gets unsandboxed, under a setting named "max". That contradicted the "upper bound, never a grant" wording this branch itself introduced.

docker --ulimit nofile=N sets soft=hard=N regardless of the parent, so the current behaviour is the stricter reading, not the conventional one. Callers who genuinely need a bigger budget raise it on sandlock itself (prlimit, systemd LimitNOFILE=), after which the clamp stops constraining. If you would rather match container convention, say so and I will drop the soft clamp along with the "never a grant" sentence.

2. The cap is not one-way under a privileged supervisor.

Nothing here drops CAP_SYS_RESOURCE, so a child under a root sandlock raises the cap straight back — verified: 64 becomes 4096 and 300 descriptors open. Unlike Landlock and seccomp, which stay irreversible for root, this is a resource budget rather than confinement, and the comment, rustdoc, reference table and Python docstring now say so.

Doing better would mean PR_CAPBSET_DROP(CAP_SYS_RESOURCE) plus capset, which is a new subsystem in a crate that touches no capabilities today, and it would change behaviour for every privileged sandbox rather than only this flag. Out of scope here; happy to take it separately if you want it.

Zero

Rejected at build() rather than per binding. It cannot be honoured — the child needs descriptors to reach main, so it would die with EMFILE from the loader and exit 127 with nothing naming the setting. The check sits next to the existing max_cpu one, which already rejects zero the same way. Only the Go SDK filters zero itself, and it has to: a uint32 struct field cannot express "unset" any other way.

Tests

Three of them exist because the suite could not see the defects review found:

• the cap is applied after Landlock and seccomp — moving setrlimit earlier makes the child die before the notify fd is read, which the original test tolerated;
• the cap never raises an inherited soft limit — invisible on hosts where soft equals hard, as CI's are;
• the enforcement test branches on geteuid(), which also fixes a real defect: the original failed under root on a correct build, because it asserted the raise is always denied.

The chroot path gets its own test: a failure there surfaces as EIO from exec-fd injection, not the EMFILE the docs first claimed. The measured floor under chroot is 13, not 17 as first written.

Also corrected the Go doc comment, which promised RLIMIT_NOFILE while nothing enforced it.

Adjacent things I did not touch

While working through this I noticed a few similar shapes elsewhere — settings where zero means two different things depending on the surface, and places where an invalid argument is coerced into a valid one rather than refused. I would rather not assert any of it from a PR description without reproductions, so I am checking them one at a time and will open a separate issue with the analysis once they are verified.

Three specific follow-ups from this change, all deliberately out of scope: dropping CAP_SYS_RESOURCE for privileged runs, mapping a failed SECCOMP_ADDFD to EMFILE rather than EACCES/EIO, and a test for the in-process entrypoint under a cap.

[congwang-mk]

On the two decisions flagged for explicit attention:

1. The soft clamp: keep it as written. For a sandboxing tool the strict reading is the right one: a setting named "max" should only ever cap, never widen the guest's budget past what the same command gets unsandboxed. The docker convention makes sense for a general-purpose runtime, less so here. The escape hatch is legitimate (raise the limit on sandlock itself via prlimit or LimitNOFILE=), and the behavior is documented and pinned by a test. Keep the "upper bound, never a grant" wording too.

2. CAP_SYS_RESOURCE: agreed, leave it alone. Dropping it would change behavior for every privileged sandbox, and a root guest that legitimately relies on the capability (for other rlimits, quotas, etc.) would break silently. That is a bigger decision than this flag and I don't want to take it as a side effect here. Documenting the cap as a resource budget rather than confinement, and splitting the enforcement test on euid so it stays honest under sudo, is the right call.