[yuravk]

Summary

Runs the compose tests for ppc64le and s390x in 4 AlmaLinux containers
(podman, systemd as PID 1), split between them the same way as on
x86_64 and aarch64.

• AlmaLinux organization: natively on POWER runners
  (ubuntu-26.04-ppc64le, -p10 for AlmaLinux 10 and Kitten) and on
  ubuntu-26.04-s390x.
• Forks: under QEMU user-mode emulation on an x86_64 runner. The
  tests' time limits are scaled ×8 there instead of ×2, and the tests
  get emulated_guest=true.

The containers share the runner's kernel, so the tests run with the tmt
context container=yes and CONTAINERTEST=1.

Changes

CI

• ci/container-guests.sh starts the containers and waits for systemd
  to boot in each one.
  • Under emulation, units with credentials fail with
    243/CREDENTIALS. The script runs sysusers and tmpfiles itself,
    gives journald a copy of its unit without credentials, and clears
    the credentials of libvirtd and virtsecretd (libvirt 12.8, Kitten).
  • It checks that the containers resolve names. When they don't, it
    prints what DNS goes through, reloads podman's network once, and
    fails the step if that doesn't help.
• Container network: if podman's default network (10.88.0.0/16)
  overlaps the runner's routes or nameservers, the containers get a
  network of their own on a free subnet. One native s390x runner's DNS
  server was 10.88.92.1, so the containers' queries went into the
  bridge and no name resolved.
• Native s390x runner: it's an Incus container with AppArmor. A
  program started from a path one of its profiles attaches to (e.g.
  /usr/bin/hostname, /usr/bin/gs) failed to load libc, so the
  confining profiles are unloaded. The unconfined ones are kept: podman
  needs them.
• inotify instances: the rootful containers share root's inotify
  instances. The limit is raised to 8192 where the runner allows it.
  The native runners don't allow it and keep 128 or 1024.
  ci/inotify-peak.sh logs the peak count while the tests run, and
  which process and container hold them. With 4 containers the peak is
  46–48, against a limit of 128. In 2 earlier runs something held about
  75 more for over 15 minutes, and D-Bus and libvirtd failed; this log
  is there to catch it if it happens again.
• Container setup: curl-minimal is replaced with curl, hostname is
  installed, and so are rsync (AlmaLinux 10) and initscripts
  (AlmaLinux 8), so the containers are closer to an installed system.
• Skipped tests: the run-page summary and the Mattermost message
  list the skipped tests with the reason.

Tests

• Disabled for container=yes:
  • attr: needs loop-device mounts.
  • libbpf-tools and perf: need BPF and perf events of the runner's
    kernel.
  • libvirt/net: needs libvirt networks.
  • podman: needs nested containers.
  • selinux: needs SELinux.
  • sysstat/iostat: drops the page cache.
• Skipped with CONTAINERTEST=1:
  • p_kvdo: needs a kernel module.
  • p_tcpdump: only the NIC check.
  • p_osbuild: an image build needs loop devices and mounts.
• s390x fixes (not product bugs):
  • p_bash: the vendor in the build triplet is ibm up to AlmaLinux 9
    and redhat from 10.
  • libbpf-tools: bpf-numamove isn't shipped for s390x.
  • libvirt/socket: an emulated s390x container has no
    /proc/sysinfo.
  • p_gzip: z15 and newer compress in hardware (DFLTCC), so only the
    uncompressed size and the name are checked.
• Emulated guests:
  • p_exim, p_openssh: the expect helpers wait 30 seconds for an
    answer.
  • p_osbuild: skipped, because building an image takes over an hour.
• t_GetArch: takes the architecture from uname when there is no
  kernel package.

Testing

• Native: AlmaLinux 8, 9, 10 and Kitten pass on both ppc64le and
  s390x. Jobs take 11–25 minutes.
• Emulated (fork): AlmaLinux 8, 9, 10 and Kitten pass on both
  architectures.
• Not yet run: the separate network. The runners used since the fix
  don't overlap 10.88.0.0/16, so they kept podman's default network as
  before.