RFC: toolchain for building eBPF modules within the OpenWrt build system

[Rosen Penev]

On Sun, Oct 3, 2021 at 5:47 AM Felix Fietkau <nbd at nbd.name> wrote:
>
>
> Hi,
>
> I recently spent some time digging into what's needed for proper eBPF
> build support in OpenWrt. Here's what I found so far:
>
> Most out-of-tree eBPF based projects fork some of the BPF related kernel
> headers from various different kernel versions and manually maintain
> those forks. These header files are usually very incomplete and tailored
> specifically for the project that uses them. To make things even worse,
> they typically explicitly rely on including headers from the local x86
> host toolchain header files when building for the BPF target.
>
> The in-kernel build of BPF modules is weird in a different way. It
> explicitly includes all the header files from the arch that the kernel
> is being built for. And because a lot of assembly stuff in there is
> completely incompatible with building for the BPF target, the build
> system actually targets clang to the same arch that the kernel is being
> built for and only emits un-optimized LLVM bitcode without running any
> of the LLVM passes. It then passes that bitcode to the optimizer and
> target code generator with bpf specified as target arch.
> It's a weird hack, but it seems to work properly even when
> cross-compiling to non-x86 targets.
>
> When writing ebpf modules myself, I definitely don't want to rely on the
> crappy header fork mess that most out-of-tree projects use. In my tests
> it was simply too fragile, and I couldn't get it to work on my macOS
> build host either. Even if we could make it work for more use cases, it
> would still be a maintenance hell when comes to supporting more features
> and newer kernel versions.
>
> I think staying close to the way that in-tree BPF module builds work is
> the way to go. Unfortunately, this means that we will not be able to use
> GCC for BPF (which Daniel has been working on), because targeting the
> frontend and the rest of the compiler to different architectures is only
> supported by clang/LLVM.
>
> I just did a test build of LLVM with reduced features and targetting
> only BPF, and it takes around 42 minutes to build on my 2018 macbook.
> That's more than the time needed to build a typical basic OpenWrt build
> from scratch.
>
> The way I see it, these are our options for eBPF support:
>
> 1. Add a host dependency on a recent enough LLVM version for eBPF.
> 1a) disable it by default
> 1b) hide eBPF packages unless host support is available
> 2. Add llvm to tools/ to ensure that it is only built once, even when
> switching between targets
> 3. Add llvm as a host package and use build-dependencies in eBPF
> 4. Ship precompiled big and little endian eBPF binaries + scripts to
> recompile in packages, and rely on CO-RE (compile once, run everywhere)
>
> What do you think?
I've been thinking of adding a toolchain/llvm section before. I don't
know how realistic it would be or if it solves the eBPF issue.
>
> - Felix
>
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel at lists.openwrt.org
> https://lists.openwrt.org/mailman/listinfo/openwrt-devel