Skip to content

ebpftracer: report missing BPF tracing program types clearly - #367

Open
KR-Ravindra wants to merge 1 commit into
coroot:mainfrom
KR-Ravindra:fix/bpf-tracing-program-type-check
Open

ebpftracer: report missing BPF tracing program types clearly#367
KR-Ravindra wants to merge 1 commit into
coroot:mainfrom
KR-Ravindra:fix/bpf-tracing-program-type-check

Conversation

@KR-Ravindra

@KR-Ravindra KR-Ravindra commented Sep 8, 2026

Copy link
Copy Markdown

Problem

On kernels built without CONFIG_BPF_EVENTS (for example NVIDIA JetPack 5 / L4T 5.10.120-tegra, where CONFIG_KPROBES=n and CONFIG_UPROBE_EVENTS=n), the agent passes every pre-flight check (tracefs present, kernel version >= 5.1, collection spec loads) and then exits with:

failed to load collection: program sched_process_exit: load program: invalid argument

Nothing in the message points at the kernel configuration, the program name varies between runs (it is whichever program cilium/ebpf loaded first), and the *ebpf.VerifierError branch prints nothing because this is not a verifier error. Users end up debugging BTF, kernel version or verifier issues that are not the cause.

Root cause

ebpftracer/tracer.go:297-305 (ebpf.NewCollectionWithOptions) receives a bare EINVAL from bpf(BPF_PROG_LOAD). With CONFIG_BPF_EVENTS off, BPF_PROG_TYPE_TRACEPOINT and BPF_PROG_TYPE_KPROBE are compiled out of the kernel and find_prog_type() returns -EINVAL. Every program in the agent's collection is a tracepoint/, kprobe/ or uprobe/ program (uprobes are BPF_PROG_TYPE_KPROBE), so nothing can load, but the agent has no check that names this condition.

Fix

  • ebpftracer/tracer.go: before loading the collection, probe ebpf.TracePoint and ebpf.Kprobe with features.HaveProgramType from the already-required github.com/cilium/ebpf module (it maps the probe's EINVAL to ebpf.ErrNotSupported). On a conclusive ErrNotSupported the agent now fails with:

    kernel does not support BPF TracePoint programs (CONFIG_BPF_EVENTS is not set?): not supported
    

    The error wraps ebpf.ErrNotSupported itself rather than the probe error: features.HaveProgramType returns an *ebpf.UnsupportedFeatureError whose text is TracePoint not supported (requires >= v4.7), and that kernel-version hint is misleading on a 5.10 kernel that simply has the option disabled. Any other probe result (for example EPERM under kernel lockdown) is ignored so the real collection load surfaces the same error it does today. The probe sits behind a package-level function variable so the mapping can be unit-tested without root or the VM=1 harness. Cost on the happy path is two tiny program loads, cached by the features package.

  • README.md: one line stating that the kernel must be built with CONFIG_BPF_EVENTS=y.

No new dependencies; go.mod/go.sum are unchanged.

How tested

New ebpftracer/tracer_progtype_test.go (TestCheckProgramTypes) injects a fake probe and checks: supported -> no error; a probe error shaped like the real *ebpf.UnsupportedFeatureError (Kprobe not supported (requires >= v4.1): not supported) -> the returned error wraps ebpf.ErrNotSupported and its text is exactly kernel does not support BPF Kprobe programs (CONFIG_BPF_EVENTS is not set?): not supported; inconclusive probe error -> no error (falls through to the real load).

The exact-string assertion catches the kernel-version tail. With %w applied to the probe error instead of the sentinel (the first revision of this PR):

--- FAIL: TestCheckProgramTypes/not_supported (0.00s)
    tracer_progtype_test.go:33: Error message not equal:
        expected: "kernel does not support BPF Kprobe programs (CONFIG_BPF_EVENTS is not set?): not supported"
        actual  : "kernel does not support BPF Kprobe programs (CONFIG_BPF_EVENTS is not set?): Kprobe not supported (requires >= v4.1): not supported"
FAIL	github.com/coroot/coroot-node-agent/ebpftracer	0.012s

With the sentinel wrapped:

=== RUN   TestCheckProgramTypes
    --- PASS: TestCheckProgramTypes/supported (0.00s)
    --- PASS: TestCheckProgramTypes/not_supported (0.00s)
    --- PASS: TestCheckProgramTypes/inconclusive_probe (0.00s)
ok  	github.com/coroot/coroot-node-agent/ebpftracer	0.012s

The GO job from .github/workflows/ci.yml was run against this revision in a golang:1.24 container (linux/amd64, libsystemd-dev installed): gofmt -l . clean, go vet ./... OK, go test ./... OK for every package with tests (cgroup, common, ebpftracer, ebpftracer/l7, logs, node, proc), go build -mod=readonly . OK. goimports -l ./ebpftracer is clean. The existing VM=1 tracer tests were not run here; I have not been able to run the changed binary on the affected Tegra kernel from this environment, so the end-to-end message on real hardware is worth a check by anyone who has one.

Links

This change was prepared with an AI agent operated by KR-Ravindra, who reviewed and tested it.

On kernels built without CONFIG_BPF_EVENTS (e.g. NVIDIA JetPack 5 / L4T
5.10 with CONFIG_KPROBES=n), BPF_PROG_TYPE_TRACEPOINT and
BPF_PROG_TYPE_KPROBE are compiled out and bpf(BPF_PROG_LOAD) fails with a
bare EINVAL. The agent then exits with

  failed to load collection: program sched_process_exit: load program: invalid argument

where the program name is whichever program happened to load first, and
nothing points at the kernel configuration.

Probe TracePoint and Kprobe support with cilium/ebpf/features before
loading the collection and fail with an error naming the missing kernel
option. Only a conclusive ebpf.ErrNotSupported is reported; any other
probe result falls through to the real collection load so existing error
paths are unchanged. The probe is behind a package-level function
variable so the message mapping is unit-tested without root or a VM.

Document the CONFIG_BPF_EVENTS requirement in the README.
@KR-Ravindra
KR-Ravindra force-pushed the fix/bpf-tracing-program-type-check branch from 0129a76 to 3f9d9d5 Compare September 8, 2026 06:53
@KR-Ravindra

Copy link
Copy Markdown
Author

Self-review before marking ready. The agent now probes the tracing program types it needs before loading and fails with a message naming the likely kernel option instead of a bare EINVAL; only a conclusive not-supported result short-circuits, so lockdown and permission errors still take the existing path. Unit test pins the exact message; vet, tests and build pass in a golang:1.24 container with libsystemd-dev.

@KR-Ravindra
KR-Ravindra marked this pull request as ready for review September 8, 2026 06:58
@KR-Ravindra

Copy link
Copy Markdown
Author

Round 1 self-review.

Checked features.HaveProgramType in cilium/ebpf v0.20.0: probeProgram maps EINVAL/E2BIG from BPF_PROG_LOAD to ebpf.ErrNotSupported and returns anything else (for example EPERM) unchanged, so the conclusive/inconclusive split in checkProgramTypes matches the library. On a kernel with CONFIG_BPF_EVENTS=y both probes return nil and the load path is unchanged. go vet ./ebpftracer/, go mod tidy (no diff) and TestCheckProgramTypes pass here. Marking ready for review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Unhelpful fatal error on kernels built without BPF tracing program types: failed to load collection: program <name>: load program: invalid argument

1 participant