Add OCI image support

[henrybear327]

Please see the prior work: #34

The idea is that we implement basic OCI operations, such as pull, unpack, run, prune, status, policy (also the the counterparts like rmi, to complete the lifecycle so we can handle the resource clean up, etc.)

Instead of relying on pure C, this PR relying on the existing go tooling (since container ecosystems are mainly built with go) - go-containerregistry, crane, and umoci. It's about 2.8k lines of go code (about 5k lines of test go code). I think it's a lot more maintainable compared to the 38k lines of C code (this is without the full OCI lifecycle).

The main idea is that this go code is exposed as a command-line tool, handling just the OCI related operations.

Tracking issue: #31

Commands to try out

build/elfuse oci list
build/elfuse oci pull alpine:3
build/elfuse oci inspect alpine:3
build/elfuse oci list
build/elfuse oci rmi --force e7a1a92a5bfe 
build/elfuse oci list

build/elfuse-container run --entrypoint /usr/local/bin/python3 \\n    python:3.12 -c 'import json,math; print(json.dumps({"pi":round(math.pi,5),"ok":True}))'
build/elfuse oci run alpine:3 /bin/sh -c 'echo hello from elfuse'

---

Summary by cubic

Adds OCI image support via a new pure‑Go CLI elfuse-container, with pull/unpack/run over a spec‑compliant OCI image‑layout. macOS runs default to a case‑sensitive APFS sparsebundle with COW clones; CI now covers Linux layout/interop and a macOS HVF run smoke.

• New Features

  • CLI: pull, unpack, inspect, list, run, rmi, prune (supports --platform, --insecure for loopback registries only, and digest pins for offline use).
  • Store uses an OCI image‑layout via go-containerregistry; interop checked with crane, skopeo, and umoci (scripts/oci-interop.sh).
  • macOS default run path: case‑sensitive sparsebundle + per‑run COW clonefile; --plain-rootfs fallback available.
  • run resolves Entrypoint/Cmd/Env/User/WorkingDir, injects runtime /etc/{resolv.conf,hosts,hostname}, then execs elfuse --sysroot.

• Hardening & Fixes

  • Refactor: extracted a shared C launch entry (core/launch.c/.h) used by both the positional ELF path and elfuse-container run; wires UID/GID, envp, and cwd flags.
  • Renamed helper to elfuse-container; removed elfuse oci and multicall; docs updated to a two‑binary model.
  • Store updates serialized with <store>/.lock; auto‑pull only when a ref is truly absent.
  • GC and rmi race‑safe; prune never detaches a live bundle; sparsebundle mount‑point symlinks rejected; orphan COW clone reaping.
  • Unpack enforces exact tar modes regardless of umask and cleans up partial trees on failure.
  • Runtime /etc injection and named user/group lookups are rootfs‑bounded; procfs/device emulation for /dev/full (ENOSPC on write) and /dev/console; cached uname exposed for procfs.
  • CI builds elfuse-container, runs cross‑tool interop on Linux, and adds a macOS HVF run smoke; interop script normalizes manifests for skopeo; klauspost/compress bumped to v1.18.7.
  • Fixed memory leaks on option‑parse error paths by freeing launch‑flag allocations; minor env‑build leak fix.

^{Written for commit 2fd5111. Summary will update on new commits.}

[jserv]

Rebase latest main branch, resolve conflicts, and refine per review messages.

[jserv]

Commands to try out

build/elfuse oci list
build/elfuse oci pull alpine:3
build/elfuse oci inspect alpine:3
build/elfuse oci list
build/elfuse oci rmi --force e7a1a92a5bfe 
build/elfuse oci list
build/elfuse-container run --entrypoint /usr/local/bin/python3 \\n    python:3.12 -c 'import json,math; print(json.dumps({"pi":round(math.pi,5),"ok":True}))'
build/elfuse oci run alpine:3 /bin/sh -c 'echo hello from elfuse'

Leveraging existing Go-based tools and packages is a great step toward full OCI image support. I agree with this change.

I suggest repositioning the build/elfuse binary as an efficient Linux syscall-to-macOS/Darwin runtime, while implementing build/elfuse-container in Go using OCI-related packages.

In other words, build/elfuse oci should not be considered a valid command. OCI-specific functionality should reside in elfuse-container, not in the elfuse executable.

[cubic-dev-ai]

25 issues found and verified against the latest diff

Prompt for AI agents (unresolved issues)

Check if these issues are valid — if so, understand the root cause of each and fix them. If appropriate, use sub-agents to investigate and fix each issue separately.


<file name="cmd/elfuse-container/list.go">

<violation number="1" location="cmd/elfuse-container/list.go:32">
P2: Concurrent `rmi` can make `list` fail rather than produce a coherent result: it reads pins and manifests without the store lock while `rmi` deletes both under that lock. Hold the same lock across snapshot construction so lifecycle commands cannot interleave.</violation>

<violation number="2" location="cmd/elfuse-container/list.go:57">
P3: Variant-specific images are displayed as only `os/arch`, so `list` loses part of the selected OCI platform. Include `cfg.Variant` when present.</violation>
</file>

<file name="src/main.c">

<violation number="1" location="src/main.c:506">
P2: Direct `elfuse --sysroot ... --workdir relative/path` can start in a host-relative directory rather than the guest root. Reject empty or non-`/`-prefixed workdirs when parsing, matching documented and container-wrapper behavior.</violation>
</file>

<file name="cmd/elfuse-container/run.go">

<violation number="1" location="cmd/elfuse-container/run.go:121">
P2: Terminal signals reach this child twice because it remains in the helper's process group and this branch forwards the helper's copy again. Signal-aware guests can run an interrupt/termination handler twice; avoid forwarding group-delivered signals, or put the child in a separate process group and forward exactly once.</violation>
</file>

<file name="cmd/elfuse-container/sparsebundle.go">

<violation number="1" location="cmd/elfuse-container/sparsebundle.go:70">
P3: Attach failures omit hdiutil's diagnostic because `Output()` discards stderr while the error message prints `out`. Use `CombinedOutput()` so invalid-image and mount errors remain actionable.</violation>

<violation number="2" location="cmd/elfuse-container/sparsebundle.go:191">
P2: Mount paths containing XML-escaped characters are returned as literal entities, so marker creation and cleanup use a nonexistent path after attach. Decode the plist string (preferably with an XML/plist parser) before returning it.</violation>
</file>

<file name="cmd/elfuse-container/commands.go">

<violation number="1" location="cmd/elfuse-container/commands.go:130">
P1: `run --platform` is ignored whenever the ref is already present, so a previously pulled different architecture can be launched despite the requested platform. Store pins per platform or verify the pinned image platform and pull/select the requested variant before computing the run spec.</violation>
</file>

<file name="cmd/elfuse-container/inspect.go">

<violation number="1" location="cmd/elfuse-container/inspect.go:31">
P2: `inspect --json` exits successfully when stdout fails (for example a closed pipe or full redirected filesystem), so callers can consume truncated/empty JSON as success. Propagate writer errors; apply the same handling to the human-summary writes.</violation>
</file>

<file name="cmd/elfuse-container/etc.go">

<violation number="1" location="cmd/elfuse-container/etc.go:43">
P2: `injectRuntimeFiles` silently accepts when `/etc` is a regular file, leading to a confusing error later when writing runtime files inside it. If the rootfs has a regular file at `etc` (not a symlink and not a directory), `root.Lstat("etc")` succeeds without the symlink flag, so the removal branch is skipped. Then `root.Mkdir("etc", 0o755)` fails with `EEXIST` which is swallowed by the `errors.Is(err, fs.ErrExist)` check. The code proceeds believing `/etc` is a usable directory, but the subsequent `writeRuntimeFile(root, "etc/hostname", ...)` calls fail with a "not a directory" error — confusing because the user never asked to remove anything.

Consider adding an explicit `li.IsDir()` check after the symlink guard: if `etc` exists and is not a directory and not a symlink, remove it first (or return a clear error) before calling `Mkdir`.</violation>

<violation number="2" location="cmd/elfuse-container/etc.go:76">
P2: When host resolver config is unavailable, guest DNS is silently redirected to Google's public resolver, which can leak private/split-DNS lookups and does not preserve the stated host-network behavior. Surface this as a run error or require an explicit fallback resolver instead of hard-coding 8.8.8.8.</violation>
</file>

<file name="cmd/elfuse-container/unpack_test.go">

<violation number="1" location="cmd/elfuse-container/unpack_test.go:263">
P3: TestUnpackHardlink's comment claims the test validates 'Both must refer to the same inode (hardlink), same content' but the test only checks that os.Stat succeeds for the hardlink entry. It doesn't verify content equality (os.ReadFile + string comparison) or inode sharing (os.SameFile on FileInfo). Consider either adding those checks or trimming the comment to match what's actually verified.</violation>

<violation number="2" location="cmd/elfuse-container/unpack_test.go:320">
P3: TestUnpackReadsExactSize comment claims to verify 'no over-read, no short read' but only validates short reads. The countingReader has exactly hdr.Size bytes, so any over-read attempt just hits EOF and r.n stays at len(content). To detect over-reads, the reader needs MORE data than Size and the test must verify only Size bytes were consumed (e.g., a reader that panics if Read is called after Size bytes). Consider updating the comment to reflect what's actually tested, or making the reader defensive.</violation>
</file>

<file name="scripts/oci-interop.sh">

<violation number="1" location="scripts/oci-interop.sh:93">
P2: Setting `PLAT_OS` or `PLAT_ARCH` makes this check fail against a correctly pulled image: those variables change only `crane manifest` selection, not the preceding `elfuse-container pull`. Keep this comparison fixed to the helper default, or pass the same platform to `pull` before comparing.</violation>
</file>

<file name="cmd/elfuse-container/gc.go">

<violation number="1" location="cmd/elfuse-container/gc.go:33">
P1: A concurrent `pull` can finish with blobs deleted by `prune` before its index descriptor is written. Serialize standalone GC's scan/delete phase with `addImage`; keep an internal locked variant for `rmi`, which already holds the lock.</violation>

<violation number="2" location="cmd/elfuse-container/gc.go:275">
P1: `rmi --force` can tear down an active container's filesystem: it force-detaches the Darwin sparsebundle or removes the plain rootfs while the guest still uses it. Track active runs or reject busy caches instead of treating `force` alone as permission to remove them.</violation>

<violation number="3" location="cmd/elfuse-container/gc.go:290">
P1: An `rmi` can make a concurrently starting cold `run` fail while it is unpacking. Serialize image reads/unpack with removal, or record active setup before descriptor/blob GC.</violation>
</file>

<file name="cmd/elfuse-container/unpack.go">

<violation number="1" location="cmd/elfuse-container/unpack.go:115">
P1: Valid layers with an opaque marker after child entries lose those same-layer children. Apply opaque whiteouts before processing additions for that layer, independent of tar ordering.</violation>

<violation number="2" location="cmd/elfuse-container/unpack.go:118">
P2: An invalid `.wh.` entry deletes its containing directory rather than failing extraction. Validate the whiteout suffix before joining it to the parent path.</violation>

<violation number="3" location="cmd/elfuse-container/unpack.go:228">
P2: A directory entry cannot replace a lower-layer symlink or regular file, so affected images unpack a different filesystem tree. Remove a pre-existing non-directory before creating the directory, while preserving existing real directories and their children.</violation>
</file>

<file name="Makefile">

<violation number="1" location="Makefile:157">
P3: The `oci-vet` darwin cross-vet run doesn't set `GOARCH=arm64`, so on an amd64 Linux host it vets the darwin code for `darwin/amd64` instead of the actual target `darwin/arm64`. CI already pins `GOARCH=arm64` for this vet. It's unlikely to cause a real build issue (the darwin-specific files are `//go:build darwin` without an arch constraint), but adding `GOARCH=arm64` makes the local `make oci-vet` gate match CI more closely and avoids any future arch-specific vet diagnostics from slipping through.</violation>
</file>

<file name="cmd/elfuse-container/cache_darwin.go">

<violation number="1" location="cmd/elfuse-container/cache_darwin.go:188">
P1: `prune --cache --all` can detach and delete a sparsebundle while a live `--no-clone` guest is using its base rootfs. `--no-clone` creates no `run-*` marker, so this check reports the mount idle; track active mount ownership independently (and cover the pre-clone startup window) before detaching.</violation>
</file>

<file name="cmd/elfuse-container/common.go">

<violation number="1" location="cmd/elfuse-container/common.go:49">
P3: Malformed `--platform` values are accepted and passed to `crane` instead of returning the advertised parse error. Validate nonempty components and reject inputs with more than three slash-separated segments.</violation>
</file>

<file name="cmd/elfuse-container/sparsebundle_test.go">

<violation number="1" location="cmd/elfuse-container/sparsebundle_test.go:117">
P3: TestCSMountCloseSuccessRootfsDirAndDetachAfterAttachError tests rootfsDir(), Close() lifecycle, and detachAfterAttachError() as separate behaviors in a single function. Splitting these into targeted subtests or separate functions would improve isolation and make failures easier to diagnose — a rootfsDir failure currently masks all Close and detachAfterAttachError checks.</violation>
</file>

<file name="cmd/elfuse-container/csrun.go">

<violation number="1" location="cmd/elfuse-container/csrun.go:76">
P1: Concurrent `oci run` calls for the same image can tear down each other's sysroot: the later call force-detaches the digest's already-mounted sparsebundle before creating its clone. Serialize ownership of a digest mount or reject/wait when `hasLiveClone` reports an active run, rather than treating every existing mount as stale.</violation>
</file>

<file name="cmd/elfuse-container/clone_sweep.go">

<violation number="1" location="cmd/elfuse-container/clone_sweep.go:54">
P2: hasLiveClone returns false on ReadDir errors, which could cause an unsafe volume detach during prune. If the mount directory is temporarily unreadable (e.g., transient FS issue, permission flake) while a live container run is still using it, sweepCSBundle will force-detach the sparsebundle volume out from under the running guest, causing rootfs I/O failures. `reapOrphanClones` has the same silent-fail pattern but only affects reporting, not safety.

**Suggestion:** Return `true` when `os.ReadDir` fails so the callee conservatively treats the volume as busy, matching the defensive pattern used in `processAlive` (non-ESRCH → alive).</violation>
</file>

<sub>Reply with feedback, questions, or to request a fix.

Re-trigger cubic</sub>

[cubic-dev-ai]

P1: run --platform is ignored whenever the ref is already present, so a previously pulled different architecture can be launched despite the requested platform. Store pins per platform or verify the pinned image platform and pull/select the requested variant before computing the run spec.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At cmd/elfuse-container/commands.go, line 130:

<comment>`run --platform` is ignored whenever the ref is already present, so a previously pulled different architecture can be launched despite the requested platform. Store pins per platform or verify the pinned image platform and pull/select the requested variant before computing the run spec.</comment>

<file context>
@@ -0,0 +1,225 @@
+	if err != nil {
+		return err
+	}
+	img, err := s.image(ref)
+	if err != nil {
+		// Auto-pull only when the ref is simply absent, so `run` is
</file context>

[cubic-dev-ai]

P1: prune --cache --all can detach and delete a sparsebundle while a live --no-clone guest is using its base rootfs. --no-clone creates no run-* marker, so this check reports the mount idle; track active mount ownership independently (and cover the pre-clone startup window) before detaching.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At cmd/elfuse-container/cache_darwin.go, line 188:

<comment>`prune --cache --all` can detach and delete a sparsebundle while a live `--no-clone` guest is using its base rootfs. `--no-clone` creates no `run-*` marker, so this check reports the mount idle; track active mount ownership independently (and cover the pre-clone startup window) before detaching.</comment>

<file context>
@@ -0,0 +1,195 @@
+		return nil, false, nil
+	}
+	reaped = reapOrphanClonesFn(mnt)
+	if hasLiveCloneFn(mnt) {
+		return reaped, true, nil
+	}
</file context>

[cubic-dev-ai]

P1: Concurrent oci run calls for the same image can tear down each other's sysroot: the later call force-detaches the digest's already-mounted sparsebundle before creating its clone. Serialize ownership of a digest mount or reject/wait when hasLiveClone reports an active run, rather than treating every existing mount as stale.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At cmd/elfuse-container/csrun.go, line 76:

<comment>Concurrent `oci run` calls for the same image can tear down each other's sysroot: the later call force-detaches the digest's already-mounted sparsebundle before creating its clone. Serialize ownership of a digest mount or reject/wait when `hasLiveClone` reports an active run, rather than treating every existing mount as stale.</comment>

<file context>
@@ -0,0 +1,161 @@
+// intra-volume only), so it is instant and free until the guest writes (COW).
+// It isolates each run's mutations from the warm base, so re-runs stay clean.
+func runCaseSensitive(cf commonFlags, s *store, ref, digest string, cfg *v1.ConfigFile, rf runFlags, tail []string) error {
+	m, baseRootfs, err := ensureCaseSensitiveRootfsForRun(cf, s, ref, digest, rf.sparseSize)
+	if err != nil {
+		return err
</file context>

[cubic-dev-ai]

P1: An rmi can make a concurrently starting cold run fail while it is unpacking. Serialize image reads/unpack with removal, or record active setup before descriptor/blob GC.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At cmd/elfuse-container/gc.go, line 290:

<comment>An `rmi` can make a concurrently starting cold `run` fail while it is unpacking. Serialize image reads/unpack with removal, or record active setup before descriptor/blob GC.</comment>

<file context>
@@ -0,0 +1,304 @@
+	// descriptor, which a retried rmi resolves and finishes
+	// (RemoveDescriptors is a filter, so re-removing is a no-op).
+	if lastPin {
+		if err := s.removeManifestDescriptor(digest); err != nil {
+			return rmReport{}, fmt.Errorf("rmi: remove manifest descriptor for %q: %w", ref, err)
+		}
</file context>

[cubic-dev-ai]

P3: TestCSMountCloseSuccessRootfsDirAndDetachAfterAttachError tests rootfsDir(), Close() lifecycle, and detachAfterAttachError() as separate behaviors in a single function. Splitting these into targeted subtests or separate functions would improve isolation and make failures easier to diagnose — a rootfsDir failure currently masks all Close and detachAfterAttachError checks.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At cmd/elfuse-container/sparsebundle_test.go, line 117:

<comment>TestCSMountCloseSuccessRootfsDirAndDetachAfterAttachError tests rootfsDir(), Close() lifecycle, and detachAfterAttachError() as separate behaviors in a single function. Splitting these into targeted subtests or separate functions would improve isolation and make failures easier to diagnose — a rootfsDir failure currently masks all Close and detachAfterAttachError checks.</comment>

<file context>
@@ -0,0 +1,349 @@
+	}
+}
+
+func TestCSMountCloseSuccessRootfsDirAndDetachAfterAttachError(t *testing.T) {
+	oldDetach := detachForce
+	var detached []string
</file context>

[cubic-dev-ai]

P3: TestUnpackHardlink's comment claims the test validates 'Both must refer to the same inode (hardlink), same content' but the test only checks that os.Stat succeeds for the hardlink entry. It doesn't verify content equality (os.ReadFile + string comparison) or inode sharing (os.SameFile on FileInfo). Consider either adding those checks or trimming the comment to match what's actually verified.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At cmd/elfuse-container/unpack_test.go, line 263:

<comment>TestUnpackHardlink's comment claims the test validates 'Both must refer to the same inode (hardlink), same content' but the test only checks that os.Stat succeeds for the hardlink entry. It doesn't verify content equality (os.ReadFile + string comparison) or inode sharing (os.SameFile on FileInfo). Consider either adding those checks or trimming the comment to match what's actually verified.</comment>

<file context>
@@ -0,0 +1,348 @@
+	}
+}
+
+func TestUnpackHardlink(t *testing.T) {
+	root, dir := newRoot(t)
+	applyEntryWithContent(t, root, regHeader("etc/passwd", 0o644, 5), "hello")
</file context>

[cubic-dev-ai]

P3: TestUnpackReadsExactSize comment claims to verify 'no over-read, no short read' but only validates short reads. The countingReader has exactly hdr.Size bytes, so any over-read attempt just hits EOF and r.n stays at len(content). To detect over-reads, the reader needs MORE data than Size and the test must verify only Size bytes were consumed (e.g., a reader that panics if Read is called after Size bytes). Consider updating the comment to reflect what's actually tested, or making the reader defensive.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At cmd/elfuse-container/unpack_test.go, line 320:

<comment>TestUnpackReadsExactSize comment claims to verify 'no over-read, no short read' but only validates short reads. The countingReader has exactly hdr.Size bytes, so any over-read attempt just hits EOF and r.n stays at len(content). To detect over-reads, the reader needs MORE data than Size and the test must verify only Size bytes were consumed (e.g., a reader that panics if Read is called after Size bytes). Consider updating the comment to reflect what's actually tested, or making the reader defensive.</comment>

<file context>
@@ -0,0 +1,348 @@
+
+// Ensure applyEntry reads exactly the header's Size bytes from the reader
+// (no over-read, no short read).
+func TestUnpackReadsExactSize(t *testing.T) {
+	root, dir := newRoot(t)
+	content := "abc123"
</file context>

[cubic-dev-ai]

P3: The oci-vet darwin cross-vet run doesn't set GOARCH=arm64, so on an amd64 Linux host it vets the darwin code for darwin/amd64 instead of the actual target darwin/arm64. CI already pins GOARCH=arm64 for this vet. It's unlikely to cause a real build issue (the darwin-specific files are //go:build darwin without an arch constraint), but adding GOARCH=arm64 makes the local make oci-vet gate match CI more closely and avoids any future arch-specific vet diagnostics from slipping through.

Prompt for AI agents

Check if this issue is valid — if so, understand the root cause and fix it. At Makefile, line 157:

<comment>The `oci-vet` darwin cross-vet run doesn't set `GOARCH=arm64`, so on an amd64 Linux host it vets the darwin code for `darwin/amd64` instead of the actual target `darwin/arm64`. CI already pins `GOARCH=arm64` for this vet. It's unlikely to cause a real build issue (the darwin-specific files are `//go:build darwin` without an arch constraint), but adding `GOARCH=arm64` makes the local `make oci-vet` gate match CI more closely and avoids any future arch-specific vet diagnostics from slipping through.</comment>

<file context>
@@ -125,6 +126,55 @@ elfuse: $(ELFUSE_BIN)
+# non-darwin stubs are checked from a macOS host. oci-lint bundles both so a
+# local run matches the CI gate.
+.PHONY: oci-vet oci-fmt-check oci-lint
+oci-vet:
+	$(Q)$(GO) vet ./cmd/elfuse-container/
+	$(Q)GOOS=darwin $(GO) vet ./cmd/elfuse-container/
</file context>

[henrybear327]

Putting this PR as draft for now. Experimenting with Claude code and the setup went wrong, the PoC code polluted the codebase.

[jserv]

Refine per review messages from @cubic-dev-ai and squash commits for further review.

[henrybear327]

Refine per review messages from @cubic-dev-ai and squash commits for further review.

I am in the process of resetting the changes.

The original 8 commits would be amended for changes as those were the logically reviewable units.

[jserv]

The original 8 commits would be amended for changes as those were the logically reviewable units.

Avoid using Semantic Commit Messages, Instead, stick to How to Write a Git Commit Message,