RFC: toolchain for building eBPF modules within the OpenWrt build system

[Thomas Hühn]

Hiho,

Thx for the nice summary!
From my more admin, testing, running Mesh network perspective on OpenWrt I would prefer option 4)

Reasons: 
-eBPF filters running in OpenWrt should be a reviewed process overseen by the core team
-the CO-RE concept with userspace scripting for mature eBPF filters sounds like that it could be made solid
-eBPF filter development vs. its usage of a robus subset of eBPFs… I prefer the latter one

Greetings Thomas


> On 3. Oct 2021, at 14:44, Felix Fietkau <nbd at nbd.name> wrote:
> 
> 
> Hi,
> 
> I recently spent some time digging into what's needed for proper eBPF
> build support in OpenWrt. Here's what I found so far:
> 
> Most out-of-tree eBPF based projects fork some of the BPF related kernel
> headers from various different kernel versions and manually maintain
> those forks. These header files are usually very incomplete and tailored
> specifically for the project that uses them. To make things even worse,
> they typically explicitly rely on including headers from the local x86
> host toolchain header files when building for the BPF target.
> 
> The in-kernel build of BPF modules is weird in a different way. It
> explicitly includes all the header files from the arch that the kernel
> is being built for. And because a lot of assembly stuff in there is
> completely incompatible with building for the BPF target, the build
> system actually targets clang to the same arch that the kernel is being
> built for and only emits un-optimized LLVM bitcode without running any
> of the LLVM passes. It then passes that bitcode to the optimizer and
> target code generator with bpf specified as target arch.
> It's a weird hack, but it seems to work properly even when
> cross-compiling to non-x86 targets.
> 
> When writing ebpf modules myself, I definitely don't want to rely on the
> crappy header fork mess that most out-of-tree projects use. In my tests
> it was simply too fragile, and I couldn't get it to work on my macOS
> build host either. Even if we could make it work for more use cases, it
> would still be a maintenance hell when comes to supporting more features
> and newer kernel versions.
> 
> I think staying close to the way that in-tree BPF module builds work is
> the way to go. Unfortunately, this means that we will not be able to use
> GCC for BPF (which Daniel has been working on), because targeting the
> frontend and the rest of the compiler to different architectures is only
> supported by clang/LLVM.
> 
> I just did a test build of LLVM with reduced features and targetting
> only BPF, and it takes around 42 minutes to build on my 2018 macbook.
> That's more than the time needed to build a typical basic OpenWrt build
> from scratch.
> 
> The way I see it, these are our options for eBPF support:
> 
> 1. Add a host dependency on a recent enough LLVM version for eBPF.
> 1a) disable it by default
> 1b) hide eBPF packages unless host support is available
> 2. Add llvm to tools/ to ensure that it is only built once, even when
> switching between targets
> 3. Add llvm as a host package and use build-dependencies in eBPF
> 4. Ship precompiled big and little endian eBPF binaries + scripts to
> recompile in packages, and rely on CO-RE (compile once, run everywhere)
> 
> What do you think?
> 
> - Felix
> 
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel at lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-devel