CI: kernel-matrix workflow from yeet new, one row per object and program

.github/workflows/kernel-matrix.yml

@@ -0,0 +1,226 @@
+name: kernel-matrix
+
+# Build every BPF object once per job, then boot a range of kernels and confirm
+# each one's verifier accepts every program in bin/*.bpf.o. The check is the
+# vendored static `veristat` (it loads each program and reports a verdict);
+# kernels come from cilium's little-vm-helper (quay.io/lvh-images), booted under
+# QEMU/KVM on the runner. Each job writes a detail table to its step summary and
+# uploads its result; the final `matrix` job pivots them into one ✅/❌ grid.
+#
+# httpscope links one object per bpf/<name>/ directory — socket, wire, walk,
+# and the TLS taps (ssl, ssl_ex, gotls, gotls_read, rustls) — because a uprobe
+# tap must load independently of the kernel-global probes. The matrix therefore
+# has a row per (object, program): peer_sendmsg/peer_recvmsg recur in every TLS
+# tap, so a bare program name would not be unique.
+#
+# Tune `matrix.kernel` to the kernel lines your script must support (`6.6`,
+# `bpf-next`, …); available lines live at
+# https://quay.io/repository/lvh-images/kind?tab=tags. Each line is resolved to
+# a concrete image at run time rather than using the floating `<ver>-main` tag,
+# which the action can't consume: little-vm-helper@v0.0.30 derives the VM image
+# filename by stripping a trailing *numeric* build stamp, so a `-main` tag
+# yields a name that doesn't match the file `lvh` actually unpacks and the run
+# dies with "invalid reference format". So each job looks up the newest
+# date-stamped tag (`<ver>-YYYYMMDD.HHMMSS`, which the action handles) — always
+# tracking the latest build, with no tag to bump and immune to quay's pruning of
+# old stamps.
+#
+# The floor is 6.6: the wire tap attaches with TCX, which 6.1 refuses at load
+# (zero instructions verified), and every other object loads on 6.1 — the
+# legacy socket tap names `iov_iter.__iov` (6.4+) but its reads are CO-RE
+# guarded, so that 6.4 floor is for the build host only, where vmlinux.h is
+# generated from the runner's own BTF by `make bpf`. So the gating lines are
+# the LTS and stable kernels from 6.6 up. bpf-next is run but does not gate:
+# it is a moving target, and what it shows tends to arrive in stable soon
+# after. Case in point: the walk VM's three nested bpf_loops verified in 14k
+# instructions until 7.2.7 backported a batch of verifier precision fixes
+# for bpf_loop callbacks ("backtracking shouldn't clear outer frame R1-R5
+# for callbacks" and kin), after which the same object cost 850k in `step`
+# alone and hit the 1M limit on 7.2.8 and on bpf-next. Its state now lives
+# in a per-CPU map the verifier does not track, and it verifies in under 5k.
+
+on:
+  workflow_dispatch:
+  push:
+    branches: [master, main]
+  pull_request:
+
+permissions:
+  contents: read
+
+jobs:
+  verify:
+    runs-on: ubuntu-latest
+    strategy:
+      fail-fast: false
+      matrix:
+        # Kernel lines to verify. Each is resolved to its newest date-stamped
+        # lvh image at run time (see the header).
+        kernel:
+          - '6.6'
+          - '6.12'
+          - '6.18'
+          - '7.2'
+          - 'bpf-next'
+        # bpf-next is informational: a failure there is recorded in the grid
+        # but does not fail the job, so the summary's gate ignores it.
+        include:
+          - kernel: 'bpf-next'
+            informational: true
+    name: kernel ${{ matrix.kernel }}
+    continue-on-error: ${{ matrix.informational == true }}
+    steps:
+      - uses: actions/checkout@v4
+
+      - name: Resolve newest lvh image tag
+        id: img
+        env:
+          KERNEL: ${{ matrix.kernel }}
+        run: |
+          set -euo pipefail
+          # Newest <line>-YYYYMMDD.HHMMSS tag (date-stamps sort
+          # lexicographically, so tail -1 is the most recent build).
+          newest="$(curl -sf "https://quay.io/api/v1/repository/lvh-images/kind/tag/?onlyActiveTags=true&limit=100&filter_tag_name=like:${KERNEL}-" \
+            | jq -r '.tags[].name' \
+            | grep -E "^${KERNEL}-[0-9]{8}\.[0-9]+$" | sort | tail -1)"
+          [ -n "$newest" ] || { echo "::error::no date-stamped tag found for kernel line '${KERNEL}'"; exit 1; }
+          echo "resolved ${KERNEL} -> ${newest}"
+          echo "tag=${newest}" >> "$GITHUB_OUTPUT"
+
+      - name: Build BPF objects + stage veristat
+        run: |
+          set -euo pipefail
+          # Builds every bin/<name>.bpf.o with the vendored static toolchain,
+          # also populating the per-machine toolchain cache
+          # (clang/bpftool/veristat).
+          make bpf
+
+          # Resolve the vendored static veristat the same way build/toolchain.mk
+          # does, and stage it into bin/ so the VM finds it under /host. It is
+          # fully static, so it runs in any kernel image's rootfs.
+          . build/toolchain.lock
+          arch="$(uname -m)"; [ "$arch" = arm64 ] && arch=aarch64
+          cache="${XDG_CACHE_HOME:-$HOME/.cache}/yeet/toolchain/v${TOOLCHAIN_VERSION}/${arch}"
+          if [ ! -x "$cache/veristat" ]; then
+            echo "::error::veristat is not in the pinned toolchain (v${TOOLCHAIN_VERSION}). Bump build/toolchain.lock to a toolchain release that ships veristat."
+            exit 1
+          fi
+          install -Dm755 "$cache/veristat" bin/veristat
+          file bin/veristat bin/*.bpf.o
+
+      - name: Verify on kernel ${{ matrix.kernel }}
+        uses: cilium/little-vm-helper@v0.0.30
+        with:
+          test-name: veristat-${{ matrix.kernel }}
+          image: kind
+          image-version: ${{ steps.img.outputs.tag }}
+          host-mount: ${{ github.workspace }}
+          install-dependencies: 'true'
+          cmd: |
+            cd /host
+            OUT_CSV=/host/.kmatrix/result.csv sh build/verify-kernel.sh
+
+      - name: Render kernel summary
+        if: always()
+        env:
+          KVER: ${{ matrix.kernel }}
+          KCSV: ${{ github.workspace }}/.kmatrix/result.csv
+        run: |
+          python3 - <<'PY' >> "$GITHUB_STEP_SUMMARY"
+          import csv, os
+          kver, path = os.environ["KVER"], os.environ["KCSV"]
+          if not os.path.exists(path):
+              print(f"### kernel `{kver}` — ⚠️ no result (build or boot failed)\n")
+              raise SystemExit
+          rows = list(csv.DictReader(open(path)))
+          mark = lambda v: "✅" if v == "success" else "❌"
+          ok = all(r["verdict"] == "success" for r in rows)
+          head = "✅ all programs loaded" if ok else "❌ verifier rejected a program"
+          print(f"### kernel `{kver}` — {head}\n")
+          print("| Object | Program | Verdict | Insns | States |")
+          print("|---|---|:---:|--:|--:|")
+          for r in rows:
+              print(f"| `{r['file_name']}` | `{r['prog_name']}` | {mark(r['verdict'])} | {r['total_insns']} | {r['total_states']} |")
+          print()
+          PY
+
+      - name: Upload result
+        if: always()
+        uses: actions/upload-artifact@v4
+        with:
+          name: kmatrix-${{ matrix.kernel }}
+          path: ${{ github.workspace }}/.kmatrix/result.csv
+          if-no-files-found: ignore
+
+  matrix:
+    needs: verify
+    if: always()
+    runs-on: ubuntu-latest
+    name: matrix summary
+    steps:
+      - uses: actions/download-artifact@v4
+        with:
+          path: results
+          pattern: kmatrix-*
+
+      - name: Render matrix
+        run: |
+          python3 - <<'PY' >> "$GITHUB_STEP_SUMMARY"
+          import csv, glob, os, re
+
+          # One CSV per kernel under results/kmatrix-<kernel>/result.csv.
+          # Rows are keyed by (object, program): peer_sendmsg/peer_recvmsg
+          # recur across the TLS taps, so a program name alone is not unique.
+          data, kernels, progs = {}, [], []
+          for d in sorted(glob.glob("results/kmatrix-*")):
+              kver = os.path.basename(d)[len("kmatrix-"):]
+              f = os.path.join(d, "result.csv")
+              if not os.path.exists(f):
+                  data[kver] = None
+                  kernels.append(kver)
+                  continue
+              data[kver] = {(r["file_name"], r["prog_name"]): r["verdict"] for r in csv.DictReader(open(f))}
+              kernels.append(kver)
+              for p in data[kver]:
+                  if p not in progs:
+                      progs.append(p)
+
+          # Order kernels by version, bpf-next last.
+          def keyf(k):
+              m = re.match(r"(\d+)\.(\d+)", k)
+              return (1, 0, 0) if not m else (0, int(m.group(1)), int(m.group(2)))
+          kernels.sort(key=keyf)
+          short = lambda k: re.sub(r"-(main|\d{8}\.\d+)$", "", k)
+
+          print("## 🐧 Kernel verification matrix\n")
+          if not progs:
+              print("⚠️ No results were produced — check the per-kernel job logs.\n")
+              raise SystemExit
+          print("| Object | Program | " + " | ".join(short(k) for k in kernels) + " |")
+          print("|---|---|" + "|".join(":-:" for _ in kernels) + "|")
+          fail = 0
+          for obj, prog in progs:
+              cells = []
+              for k in kernels:
+                  d = data[k]
+                  if d is None or (obj, prog) not in d:
+                      cells.append("⚪")
+                  elif d[(obj, prog)] == "success":
+                      cells.append("✅")
+                  else:
+                      cells.append("❌"); fail += 1
+              print(f"| `{obj}` | `{prog}` | " + " | ".join(cells) + " |")
+          print()
+          print("✅ accepted · ❌ rejected · ⚪ not run · bpf-next is informational and does not gate\n")
+          total = len(progs) * len([k for k in kernels if data[k] is not None])
+          verb = "all programs loaded on every kernel" if fail == 0 else f"{fail} of {total} program×kernel checks failed"
+          print(f"**{len(progs)} program(s) × {len(kernels)} kernel(s) — {verb}.**")
+          PY
+
+      - name: Gate on any rejection
+        run: |
+          # Fail the run if any per-kernel job failed (a rejection or a build/boot error).
+          if [ "${{ contains(needs.verify.result, 'failure') }}" = "true" ] || [ "${{ needs.verify.result }}" = "failure" ]; then
+            echo "::error::one or more kernels rejected a program (see the matrix summary)"
+            exit 1
+          fi

.gitignore

@@ -6,3 +6,6 @@ bin/*.bpf.o
 .build/
 bpf/include/vmlinux.h
 .clangd
+
+# kernel-matrix run output (build/kernel-matrix.sh)
+.kmatrix/

README.md

@@ -393,10 +393,18 @@ query overrides with `(kind)`. This is what tcpwalk2 does with a
 54 KB schema table rendered at build time from one kernel's BTF — a
 table that, checked here, had `snd_cwnd` forty bytes from where this
 kernel keeps it. The kernel side is derived from tcpwalk2's VM with
-one structural change: every op runs as a `bpf_loop` step, so the
+two structural changes. Every op runs as a `bpf_loop` step, so the
 verifier walks the dispatch once per call site instead of once per
-slot per section per field, which on this kernel was the difference
-between a million-instruction rejection and a load.
+slot per section per field. And the VM's mutable state lives in a
+one-element per-CPU array, not on the stack, with the callbacks
+reaching it by lookup and only the event pointer riding on the stack as
+the callback context: a callback loop is verified once only when its
+entry state converges with an earlier one, and the verifier does not
+track map memory, so nothing the ops do can tell one iteration from the
+last. With the state on the stack, the 7.2.7 verifier backports left
+the object simulating every one of its 8 × 16 × 8 nested iterations and
+hitting the million-instruction limit; with it in the map, the whole
+object verifies in under 5k instructions on every kernel in the matrix.
 
 The event carries the socket's 4-tuple, state and the segment's
 sequence number and lengths, read through CO-RE in a fixed prologue,
@@ -465,6 +473,28 @@ buffer held before — a response head where curl's body should be.
 - **Clean clone**: `git clone`, `npm install`, `make bpf`, `npm test`,
   `npx yeetkit build` all pass from a fresh checkout on this box. The
   yeetkit dependency is by absolute path for now.
+- **Kernel matrix**: `.github/workflows/kernel-matrix.yml` (the one
+  `yeet new` scaffolds, adapted to this repo's per-directory objects)
+  builds every `bin/*.bpf.o` on the runner, boots 6.6, 6.12, 6.18, 7.2
+  and bpf-next under cilium's little-vm-helper, and runs the vendored
+  static veristat in each VM via `build/verify-kernel.sh`. The summary
+  is a grid of (object, program) × kernel, since
+  `peer_sendmsg`/`peer_recvmsg` recur across the TLS taps. The floor is
+  6.6: 6.1 refuses the wire tap's TCX attach type at load, and every
+  other object loads there (the socket tap's `iov_iter` reads are CO-RE
+  guarded, so the 6.4 note above is for the build host only). The first
+  run also caught the 6.6 verifier rejecting the wire tap's
+  `bpf_skb_load_bytes` size as possibly zero, since a verifier before
+  6.9 does not narrow a register on a `!= 0` branch; the bound is now
+  rebuilt by arithmetic. The second catch was the walk VM: on this
+  host's 7.2.6 it verified in 14k instructions, but stable 7.2.7
+  backported a batch of verifier precision fixes for `bpf_loop`
+  callbacks, after which the lvh 7.2.8 image and bpf-next ran it to the
+  one-million limit. Moving the VM's state into a per-CPU map (see the
+  walk VM section) brought it to under 5k everywhere. bpf-next runs but
+  does not gate. `make veristat-matrix` runs the same thing
+  locally with lvh + a static qemu (Linux, KVM, root for the VM), and
+  `make veristat` is the single-kernel check against this host.
 
 ### Known limits