Skip to content

IMAGES.md

Latest commit

 

History

History
275 lines (214 loc) · 10.5 KB

IMAGES.md

File metadata and controls

275 lines (214 loc) · 10.5 KB
Raw

Images for Legion

Legion runs containers from OCI images, the same format Docker and Kubernetes use, but it does not build them. You build images with jailer, made for Legion, or with the usual tools, push them to a registry or copy them to the nodes, and Legion pulls or loads them. This guide covers where to get FreeBSD images, how to build your own, how to get them onto a cluster, and what to expect from Linux images.

What Legion runs

  • FreeBSD images run natively: each container is a jail whose root file system is the image (DESIGN.md §6.1).
  • Linux images run on nodes started with legion daemon --linux, through FreeBSD's Linux compatibility layer. That is not a Linux kernel; see Linux images below.

When an image is published for several systems, Legion takes the freebsd/<arch> entry first, then linux/<arch> on a node started with --linux. Images for any other system are refused.

Base images

FreeBSD publishes official base images on Docker Hub. The ones Legion's own tests use:

Image What is in it Use it for
docker.io/freebsd/freebsd-runtime:15.1 A minimal userland: shared libraries, sh and basic tools Small images for programs you copy in
docker.io/freebsd/freebsd-notoolchain:15.1 The base system without compilers, with pkg Installing packages with pkg install

Match the base image's release to the nodes' FreeBSD release or an older one: a jail shares the host's kernel, and programs built for a newer release may not run on an older kernel.

Building a FreeBSD image

Build with jailer, on a FreeBSD machine. It needs nothing but the base system: no buildah, Podman or Docker. Building does not need a Legion node, and is best done on a build machine or in CI rather than on a node. Install it from its release (FreeBSD 15; for arm64, aarch64 in place of amd64):

fetch https://github.com/grepjava/jailer/releases/download/v0.5.0/jailer-0.5.0-freebsd15-amd64.tar.gz
tar xzf jailer-0.5.0-freebsd15-amd64.tar.gz
doas install -s jailer-0.5.0-freebsd15-amd64/bin/jailer /usr/local/bin/
doas install -m 444 jailer-0.5.0-freebsd15-amd64/man/man1/jailer.1 /usr/local/share/man/man1/

or, on another architecture, with Rust's cargo (pkg install rust): cargo install --git https://github.com/grepjava/jailer --tag v0.5.0.

A Containerfile is written as for Docker (jailer init writes one for a Rust, Go, Node, Python or C project, or one that installs packages). This one, from Legion's stateful-service test (crates/legion/tests/stateful/Containerfile), installs PostgreSQL and removes what the server does not need at run time:

FROM docker.io/freebsd/freebsd-notoolchain:15.1
RUN env ASSUME_ALWAYS_YES=yes pkg bootstrap -f >/dev/null \
 && env ASSUME_ALWAYS_YES=yes pkg install -y postgresql17-server >/dev/null \
 && pkg delete -fy llvm19 python312 perl5 >/dev/null \
 && rm -f /usr/local/lib/postgresql/llvmjit* \
 && pkg clean -ay >/dev/null && rm -rf /var/db/pkg/repos /var/cache/pkg/*
USER postgres

Build it and try it, as root, since RUN steps and test runs are jails:

doas jailer build --strict -t pg:17 .
doas jailer run pg:17               # in a jail set up as Legion sets one up

jailer checks the image against what Legion runs: that the USER is in the image's /etc/passwd, that the entrypoint exists and is executable, and that the stop signal is one Legion knows. It warns about pkg caches left in the image and Containerfile settings Legion ignores. --strict fails the build when a check fails; jailer check runs the checks alone, on a Containerfile or an image. A rebuild reuses the RUN steps whose inputs did not change.

With buildah or Podman

They build the same Containerfiles (pkg install buildah or pkg install podman), without jailer's checks:

podman build --network host -t registry.example/shop/pg:17 .

--network host gives the RUN steps the host's network, which pkg needs to fetch packages.

Some tips:

  • Keep images small. Remove what the program does not need at run time (compilers, scripting languages pulled in as build dependencies, pkg's caches), in the same RUN step that added it, as above. Smaller images pull faster to every node.
  • Set a user. USER works as in Docker. Legion reads the user and its groups from the image's /etc/passwd and /etc/group.
  • CMD and ENTRYPOINT work as in Docker. A Legion pod's command and args replace them, as in Kubernetes.
  • Health checks in the Containerfile are not used. Give the pod probes instead (or, from a compose file, healthcheck:, which Legion turns into probes).

Several architectures

Legion runs on amd64 and aarch64 nodes, and picks the entry for its own architecture from an image index. Build the image on (or for) each architecture and publish both under one name. jailer publishes one architecture per name; for an index use buildah manifest create and buildah manifest push --all. A single-architecture image runs only on nodes of that architecture.

Getting an image to the cluster

Through a registry

Push the image to any registry that speaks the OCI distribution API: Docker Hub, GitHub's or GitLab's container registry, Harbor, a cloud provider's registry, or one you run yourself.

doas jailer login -u ci -p "$TOKEN" registry.example
doas jailer push pg:17 registry.example/shop/pg:17

(Or podman push, for an image Podman built.)

Then name it in a pod. Each node pulls it the first time it needs it:

pod:
  containers:
    - name: db
      image: registry.example/shop/pg:17

An image name without a registry means Docker Hub, as in Docker (nginx:1.27 is docker.io/library/nginx:1.27).

What a deployment runs

When a Service or Job is deployed, the leading manager asks the registry what each tag names, once, and every task of that deployment runs that digest, on whichever node it lands. A tag moved since changes nothing until the Service is deployed again; scaling it is not a deployment. The digests are on the Service's status (legion get service web -n shop, under status.images), and each task's status says which manifest its node ran, which for an image of several platforms is the one for that node's. A rollback runs the digests from before the update, whatever the tags name now.

So the managers need to reach the registries, with the pod's imagePullSecrets. Until a deployment's images are resolved, no task of it starts, and its status says why; tasks of the one before keep running. A reference by digest (nginx@sha256:...) and a localhost/ name are left as written. Pinning starts once every node runs a Legion that reads pins; until then, tasks run images by name, as before.

A container's pullPolicy: never keeps its image to what each node holds under that name: no registry is asked, by the managers or the node, and a node without the image refuses the task.

From a private registry

Put a Docker login in a Secret of the pod's namespace, in the format docker login writes to ~/.docker/config.json, and name it in the pod's imagePullSecrets:

apiVersion: legion/v1
kind: Secret
metadata: { name: registry-login, namespace: shop }
spec:
  text: |
    {"auths": {"registry.example": {"username": "ci", "password": "..."}}}
---
# in the Service's pod
pod:
  imagePullSecrets: [registry-login]
  containers:
    - name: db
      image: registry.example/shop/pg:17

auth (base64 of user:password) works in place of username and password; credential helpers do not. An image pulled with a login is kept private to the namespaces that pulled it: another namespace pulls it again, with its own login, and the registry decides.

From a registry without TLS

Legion reaches registries over HTTPS. An internal one that serves plain HTTP has to be named on every node, managers included, as the managers resolve tags and the nodes pull:

sysrc legion_args+=" --insecure-registry registry.lan:5000"

The flag repeats, one registry each, named as image references name them (host[:port]). Anyone on the network path can read and change what is pulled from such a registry, logins included, so keep it to networks you trust.

Without a registry

Save the image as an OCI archive and load it on each node that may run it, as root:

doas jailer export localhost/pg:17 pg.tar          # on the build machine
legion image load pg.tar --zfs-root zroot/legion   # on a node: the daemon's --zfs-root

(Or podman save --format oci-archive -o pg.tar localhost/pg:17.)

Name such images localhost/..., or give their containers pullPolicy: never: Legion never tries to pull those, so a node without the image fails the task instead of asking a registry. A cluster that can't reach a registry has to use one of the two, as resolving a tag needs the registry. The store finds an image by the name it was loaded under, not by digest, so deploying the loaded image by digest asks a registry too. (localhost:5000/..., with a port, is a registry on that port, and is pulled as any other.) Load the image on every node that may run the service, or keep the service to those nodes with a placement constraint:

placement:
  constraints: ["node.labels.images == local"]

legion node label n1 images=local sets that label. legion image load works beside a running daemon, and --name loads the image under another name than the archive gives.

Linux images

A node started with legion daemon --linux also runs Linux images, such as alpine and debian from Docker Hub, through FreeBSD's Linux compatibility layer (linux(4)). Both musl and glibc programs run; Legion gives such containers /proc, /sys, /dev/fd and /dev/shm as Linux programs expect them.

In a cluster where only some nodes run Linux images, ask for them:

placement:
  constraints: ["node.platform.linux == true"]

Many Linux images run unchanged. These do not:

  • Images that need Linux kernel features FreeBSD's layer does not provide: namespaces, cgroups, seccomp, io_uring, eBPF. Images that run containers or sandboxes of their own (Docker-in-Docker, some browsers, some build tools) are among them.
  • Programs that read Linux-only files under /proc or /sys, for example some monitoring agents, may fail or report wrong values.

If a program is in FreeBSD's ports, a FreeBSD image is the better choice: it runs natively, without the compatibility layer.

Removing images

Nodes remove images nothing has used for an hour, checked every ten minutes, then any layers no image uses. There is nothing to clean up by hand.