RFC: toolchain for building eBPF modules within the OpenWrt build system

[Felix Fietkau]

Hi,

I recently spent some time digging into what's needed for proper eBPF
build support in OpenWrt. Here's what I found so far:

Most out-of-tree eBPF based projects fork some of the BPF related kernel
headers from various different kernel versions and manually maintain
those forks. These header files are usually very incomplete and tailored
specifically for the project that uses them. To make things even worse,
they typically explicitly rely on including headers from the local x86
host toolchain header files when building for the BPF target.

The in-kernel build of BPF modules is weird in a different way. It
explicitly includes all the header files from the arch that the kernel
is being built for. And because a lot of assembly stuff in there is
completely incompatible with building for the BPF target, the build
system actually targets clang to the same arch that the kernel is being
built for and only emits un-optimized LLVM bitcode without running any
of the LLVM passes. It then passes that bitcode to the optimizer and
target code generator with bpf specified as target arch.
It's a weird hack, but it seems to work properly even when
cross-compiling to non-x86 targets.

When writing ebpf modules myself, I definitely don't want to rely on the
crappy header fork mess that most out-of-tree projects use. In my tests
it was simply too fragile, and I couldn't get it to work on my macOS
build host either. Even if we could make it work for more use cases, it
would still be a maintenance hell when comes to supporting more features
and newer kernel versions.

I think staying close to the way that in-tree BPF module builds work is
the way to go. Unfortunately, this means that we will not be able to use
GCC for BPF (which Daniel has been working on), because targeting the
frontend and the rest of the compiler to different architectures is only
supported by clang/LLVM.

I just did a test build of LLVM with reduced features and targetting
only BPF, and it takes around 42 minutes to build on my 2018 macbook.
That's more than the time needed to build a typical basic OpenWrt build
from scratch.

The way I see it, these are our options for eBPF support:

1. Add a host dependency on a recent enough LLVM version for eBPF.
1a) disable it by default
1b) hide eBPF packages unless host support is available
2. Add llvm to tools/ to ensure that it is only built once, even when
switching between targets
3. Add llvm as a host package and use build-dependencies in eBPF
4. Ship precompiled big and little endian eBPF binaries + scripts to
recompile in packages, and rely on CO-RE (compile once, run everywhere)

What do you think?

- Felix