Mount an OCI image read-only into an actor by ronlv10 · Pull Request #923 · agent-substrate/substrate

Ron Lev (ronlv10) · 2026-08-13T19:10:27Z

Fixes #783
Adds an image source to ActorTemplate's VolumeSource: a container can mount the contents of an OCI image it does not run. This is how tooling gets into images built by third parties without rebuilding them.

spec:
  containers:
  - name: sandbox
    image: docker.io/example/benchmark@sha256:...
    command: ["/ate/agent"]
    volumeMounts:
    - name: agent
      mountPath: /ate
  volumes:
  - name: agent
    image:
      reference: registry.example.com/agent@sha256:...

How it works

atelet pulls the image through the existing layer cache and records the volume's layers in the bundle's overlay spec, next to the rootfs layers. ateom composes the volume inside the bundle — the cached layers with no writable layer on top, so the mount is read-only — and the container binds it at the declared path. The volume is composed per container: containers of one actor may mount the same volume, and each gets its own mount point inside its own bundle, all backed by the same shared layers. On resume the volume is re-composed the same way.

References must be digest-pinned, the same rule as container images: a snapshot is only valid against the exact bytes it was taken with.

On micro-VMs the volume rides the same read-only virtio-fs share as the container rootfs: ateom stages each composed volume beside the rootfs on the host, and the guest binds it into the container at the declared path.

google-cla · 2026-08-13T19:10:37Z

Thanks for your pull request! It looks like this may be your first contribution to a Google open source project. Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

View this failed invocation of the CLA check for more information.

For the most up to date status, view the checks section at the bottom of the pull request.

A volume can name an OCI image; the reference must be digest-pinned and mounted by at least one container. ateapi projects it into the workload spec on its own volume type.

Benjamin Elder (BenTheElder)

huge +1 in concept, it's a fair bit of code understandably, I need to read back through it in depth. attaching one plausible looking agent finding now.

Benjamin Elder (BenTheElder) · 2026-08-13T22:43:27Z

+	// ImageDigest is the manifest digest the volume's ref resolved to, in the
+	// same form and for the same reason as OverlaySpec.ImageDigest: the GC's
+	// root-set scan protects an image by digest.
+	ImageDigest string `json:"imageDigest,omitempty"`


🤖 blocking 🔴 – Nothing roots these, so the GC can evict an image volume's layers while an actor is running on them. addSpecRoots in internal/imagecache/gc.go reads spec.ImageDigest and spec.Layers and never looks at spec.ImageVolumes, so a volume's digest never reaches RootSet.ImageDigests and its layer hexes never reach LayerHexes or LayerSets.

Both eviction guards then miss. The volume's image record fails the digest check and is retired, which drops its layers' refcount to zero, and the bundle-spec fallback that would otherwise save them is keyed on LayerHexes. The layers go while they are bind-mounted at <bundle>/volumes/<name>. A resume is the visible failure: setupImageVolumes calls FinalizeLayer on a directory that no longer exists.

The watermark makes it likelier rather than rarer — the GC fires under cache pressure, which is the state this feature encourages by adding images per actor.

addSpecRoots walking spec.ImageVolumes covers it. Worth giving each volume its own LayerSets signature rather than folding its layers into the rootfs one, since the signature is meant to match a record's exact layer list.

Good catch- added !

The bundle overlay spec gains ImageVolumes; ateom composes each at ateompath.ImageVolumeMountPath — a bind for one layer, an upperless overlay for several, FinalizeLayer first so whiteouts hold.

resolveImageVolumes pulls each mounted image through the layer cache and records its layers in the overlay spec; the OCI spec binds the composed volume read-only at the declared path.

The micro-VM runtime builds its own guest OCI spec, so image volumes are carried explicitly: ateompb.Container gains image_volume_mounts, staging binds each composed volume into the read-only share beside the rootfs, and the guest re-adds the binds at the declared paths.

A three-layer fixture pushed at test time: every layer visible, an upper layer shadows a lower one, a whiteout hides a file, writes are refused, and the volume survives suspend/resume.

Benjamin Elder (BenTheElder) · 2026-08-14T22:07:51Z

 message DurableDirVolumeMount {
-  // volume_name is the name the ActorTemplate gave the volume. It selects the
-  // per-volume directory atelet prepared for the actor on the host.
+  // volume_name is the name the ActorTemplate gave the volume.


🤖 nit 🟢 – This lost a sentence. It used to read "It selects the per-volume directory atelet prepared for the actor on the host", which is the only place that said what volume_name actually resolves to for a durable volume. The new ImageVolumeMount below doesn't need the same sentence — its source is composed in the bundle, not prepared by atelet — so this looks like collateral from writing the two messages side by side rather than an intended trim.

Done - returned it back as in main

Benjamin Elder (BenTheElder) · 2026-08-14T22:07:51Z

 	)
 }

+// The path is per-container: containers of one actor may mount the same


🤖 nit 🟢 – Two new exported helpers, and neither has a conventional doc comment: this one starts mid-thought ("The path is per-container") instead of with the identifier, and ImageVolumeMountPathInBundle below has none at all. The content is worth keeping — it explains why the path is per-container — it just needs "ImageVolumeMountPath returns ..." in front, and a line on the in-bundle variant saying when to reach for it (the caller already has a bundle path, e.g. setupImageVolumes).

Benjamin Elder (BenTheElder) · 2026-08-18T01:55:29Z

 		if err := kata.ReconstructSharedDirFromImage(ctx, c.bundleRootfs, id, c.name); err != nil {
 			return nil, fmt.Errorf("while staging overlay lower for %q: %w", c.name, err)
 		}
+		// Image volumes ride the same read-only virtiofsd share.


We're moving towards using a single share for host <> guest and overlay otherwise.

cc Lucky Abolorunke (@Oneimu)

(One share for everything, instead of per volume type with the carrier container back, for performance reasons)

As far as I understand, this is aligned with the current implementation, right?
Did you mean this as an FYI, should I drop the comment as obsolete, or something else?

removed the comment

…le-dir volume_name comment

Ron Lev (ronlv10)

Thank you Benjamin Elder (@BenTheElder) !

Ron Lev (ronlv10) · 2026-08-18T09:12:35Z

 message DurableDirVolumeMount {
-  // volume_name is the name the ActorTemplate gave the volume. It selects the
-  // per-volume directory atelet prepared for the actor on the host.
+  // volume_name is the name the ActorTemplate gave the volume.


Done - returned it back as in main

Ron Lev (ronlv10) · 2026-08-18T09:12:57Z

 	)
 }

+// The path is per-container: containers of one actor may mount the same


Ron Lev (ronlv10) · 2026-08-18T13:35:15Z

 		if err := kata.ReconstructSharedDirFromImage(ctx, c.bundleRootfs, id, c.name); err != nil {
 			return nil, fmt.Errorf("while staging overlay lower for %q: %w", c.name, err)
 		}
+		// Image volumes ride the same read-only virtiofsd share.


As far as I understand, this is aligned with the current implementation, right?
Did you mean this as an FYI, should I drop the comment as obsolete, or something else?

Benjamin Elder (BenTheElder) · 2026-08-18T17:55:30Z

+	// E2E_SANDBOX_CLASS selects the probe manifest variant; suites copy the
+	// probe's runtime, so this is what runs a suite on gVisor or micro-VM.
+	tmplName := "probe.yaml.tmpl"
+	if os.Getenv("E2E_SANDBOX_CLASS") == "microvm" {


nothing sets this, so we're not testing microvm?

we probably need to refactor e2e a bit so we don't depend on CI plumbing a bunch of options, but as a stopgap it seems OK to add this to the actions config.

Ahh I didn't realize that - added

Lucky Abolorunke (Oneimu) · 2026-08-18T18:16:51Z

+// StageImageVolume bind-mounts one composed image volume read-only at
+// <cid>/volumes/<name> under SharedDir(id), so virtiofsd exposes it to the
+// guest.
+func StageImageVolume(ctx context.Context, src, id, cid, volumeName string) error {


Suggestion on ordering: StageImageVolume and #1034's kata.BindIntoShare are the same stale-umount/mkdir/rbind skeleton (this one adds the ro remount). Rather than landing two parallel bind-into-share helpers, could we let #1034 merge first and rebase this over it? Then this function becomes BindIntoShare + the ro remount (or a small ro option on it), and the tree has one copy of the mount logic.

If your pr merge first I can to do the unification on my side instead.

I'm fine with both ways let's see if it is merged today :)

I chatted with lucky, let's land your PR first and we'll have his PR more generally consolidate virtiofsd

Benjamin Elder (BenTheElder) · 2026-08-18T21:11:32Z

      run: hack/run-e2e-kind.sh ./internal/e2e/suites/demo -v -args --no-color
+    - name: Run E2E tests (micro-VM image volumes)
+      # Same image volume e2e suite on the micro-VM runtime.
+      env:


TODO: factor this out so it's easy to test locally (not exclusive to this PR, it's the established pattern for now)

Benjamin Elder (BenTheElder) · 2026-08-18T21:16:42Z

+	}
+	RunCmdWithEnv(t, []string{"KO_CONFIG_PATH=" + root}, filepath.Join(root, "hack/run-tool.sh"), applyArgs...)
+
+	t.Cleanup(func() {


Human comment: I think we should probably stop hardcoding ate-e2e-probe as the preferable fix.

🤖 should-fix 🟡 – This deletes a fixture two suites now share, and nothing serializes them.

hack/run-e2e.sh ends in go test -v ./internal/e2e/suites/... with no -p 1, so Go runs the suite packages concurrently as separate binaries — the mutexes in namespace.go and testmain.go only coordinate within one process. Before this PR only identity used the probe; now imagevolume calls DeployProbe too, and both get a cleanup that runs kubectl delete -f over the same manifest, namespace ate-e2e-probe included.

The likely ordering is the bad one. imagevolume only reads the probe's WorkerPool and ActorTemplate to clone them into its own namespace, so it finishes early; identity runs its actors inside ate-e2e-probe for the whole test (CreateActor, SuspendActor, ResumeActor, /whoami all against probeNamespace). So imagevolume's cleanup deletes the namespace out from under identity's running actors. The reverse order fails too: a re-apply into a Terminating namespace is rejected.

The kind cluster is throwaway, so simply not deleting the shared fixture would do it. Giving DeployProbe a namespace argument so each suite gets its own copy is the other direction, and would also let identity stop hardcoding "ate-e2e-probe" alongside imagevolume's use of e2e.ProbeNamespace.

Ron Lev (ronlv10) force-pushed the actortemplate-image-volume branch 3 times, most recently from 4e11b94 to 5cb2ccc Compare August 13, 2026 19:34

Ron Lev (ronlv10) marked this pull request as ready for review August 13, 2026 19:37

Ron Lev (ronlv10) force-pushed the actortemplate-image-volume branch 3 times, most recently from 8eca1d8 to 416ac52 Compare August 13, 2026 21:49

api: add an image volume source to ActorTemplate

1a73806

A volume can name an OCI image; the reference must be digest-pinned and mounted by at least one container. ateapi projects it into the workload spec on its own volume type.

Ron Lev (ronlv10) force-pushed the actortemplate-image-volume branch from 416ac52 to 6137852 Compare August 13, 2026 21:54

Benjamin Elder (BenTheElder) reviewed Aug 13, 2026

View reviewed changes

Ron Lev (ronlv10) added 4 commits August 14, 2026 09:54

imagecache: compose image volumes read-only at bundle-local mount points

c6f58fb

The bundle overlay spec gains ImageVolumes; ateom composes each at ateompath.ImageVolumeMountPath — a bind for one layer, an upperless overlay for several, FinalizeLayer first so whiteouts hold.

atelet: resolve image volumes and bind them into the container

67ddb11

resolveImageVolumes pulls each mounted image through the layer cache and records its layers in the overlay spec; the OCI spec binds the composed volume read-only at the declared path.

e2e: exercise a multi-layer image volume end to end

33203f7

A three-layer fixture pushed at test time: every layer visible, an upper layer shadows a lower one, a whiteout hides a file, writes are refused, and the volume survives suspend/resume.

Ron Lev (ronlv10) force-pushed the actortemplate-image-volume branch from 6137852 to 33203f7 Compare August 14, 2026 07:03

Ron Lev (ronlv10) requested a review from Benjamin Elder (BenTheElder) August 14, 2026 14:27

Benjamin Elder (BenTheElder) reviewed Aug 18, 2026

View reviewed changes

Benjamin Elder (BenTheElder) self-assigned this Aug 18, 2026

Benjamin Elder (BenTheElder) added area/gvisor area/microVM labels Aug 18, 2026

Lucky Abolorunke (Oneimu) mentioned this pull request Aug 18, 2026

microvm: serve durable-dir volumes through the one kataShared virtiofsd #1034

Open

Ron Lev (ronlv10) added 2 commits August 18, 2026 12:14

review: conventional doc comments on ateompath helpers; restore durab…

73ddcbf

…le-dir volume_name comment

Merge agent-substrate/main into actortemplate-image-volume

d2df1d7

Ron Lev (ronlv10) requested a review from Benjamin Elder (BenTheElder) August 18, 2026 10:23

Ron Lev (ronlv10) commented Aug 18, 2026

View reviewed changes

Ron Lev (ronlv10) added 2 commits August 18, 2026 16:44

review: drop stale image-volume share comment

a9383ff

review: regenerate ateom.pb.go for the restored proto comment

bd4000f

Benjamin Elder (BenTheElder) reviewed Aug 18, 2026

View reviewed changes

Lucky Abolorunke (Oneimu) reviewed Aug 18, 2026

View reviewed changes

ci: run the imagevolume e2e suite on the micro-VM runtime

75b26c0

Benjamin Elder (BenTheElder) reviewed Aug 18, 2026

View reviewed changes

Uh oh!

Conversation

Ron Lev (ronlv10) commented Aug 13, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

How it works

Uh oh!

google-cla Bot commented Aug 13, 2026

Uh oh!

Benjamin Elder (BenTheElder) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Ron Lev (ronlv10) left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Ron Lev (ronlv10) commented Aug 13, 2026 •

edited

Loading