Revisiting ABI_VERSION handling policy

[Felix Fietkau]

On 2021-02-09 13:00, Jo-Philipp Wich wrote:
> Hi,
> 
>> The goal of reducing unnecessary build churn makes perfect sense to me,
>> but I think we need to reconsider the trade-off we're making here, and
>> the failure modes of each option.
> 
> the motivation wasn't about build-churn at all but to ensure that library
> packages can be cleanly upgraded and that different versions of libraries can
> coexist on the same system without trashing userspace mid-upgrade. The other
> major consideration is hard package version dependencies to ensure that
> library users actually depend on the binary library ABI version they've been
> linked against at build time.
Ah, that wasn't really clear to me from reading the commit I pointed out.

>> I believe that having to manually consider whether the ABI changed when
>> updating a package comes with a nasty foot-gun that can deliver very
>> subtle and hard to debug failure cases.
> 
> Upgrading a very central library package like wolfSSL which is used by uclient
> (and thus opkg), wpad (wireless connectivity) and uhttpd through ustream
> (webinterface access) is a sensible operation. In my view, sanely maintaining
> the ABI_VERSION is simply part of the work when dealing with such a package.
> 
> Throwing all that over board just because you've been annoyed by a bug that
> cost you a few hours is not the right way forward. By that metric, we
> should've dropped most of buildroot years ago already.
This has nothing to do with being annoyed by that particular bug. The
issue that I'm annoyed about is how easy it is for these things to creep
in, even with careful checks.

>> To fix this situation, I would like to switch wolfssl and maybe a few
>> other packages prone to ABI breakage back to version based ABI_VERSION.
>> At least wolfssl also needs a short suffix generated from config vars.
> 
> It makes little sense to peg the ABI_VERSION to the package version, at least
> without revising the packaging at the same time. The ABI version should only
> change if the shipped shared objects become incompatible (or more
> specifically, if the SONAME changes).
The last stable update didn't change SONAME, but it included changes to
the same data structure that triggered the bug that I ran into. It could
very likely have introduced very similar subtle breakage.
Looking at the abi-laboratory timeline, there have been many incremental
wolfssl releases that did not change the SONAME, but introduced
potentially ABI incompatible changes. In fact, it seems to me that there
are more breaking than non-breaking updates.
If we were to peg the ABI_VERSION to the package version, we could still
deliver security updates by backporting relevant patches and review
potential ABI isuses of those patches.

> By tying the ABI version to the package version without also ensuring that the
> shipped .so files are differently named, you'll also introduce file level
> package conflicts. A sanely managed ABI_VERSION should ensure that it is
> possible for libfoo1 (ABI_VERSION 1) and libfoo2 (ABI_VERSION 2) to coexist on
> the system which is a precondition for being able to cleanly opkg-upgrading
> library packages without trashing the currently running userspace.
> 
> Especially for libraries with frequent security updates like wolfSSL, I'd
> rather retain this ability.
> 
>> If we want to reduce build churn, we should find a way to automate
>> setting the ABI version, since clearly our package maintainers + review
>> process cannot (and in my opinion also should not) be trusted with this
>> extra responsibility.
> 
> It should be possible to use readelf/objdump to examine the exported symbols
> of a given shared object and see if either the size of a symbol changed or if
> previously present ones were removed - addition should be fine.
> 
> Maybe this is something that can be added to the FIXUP=1 stuff.
The breakage that I was seeing has nothing to do with exported symbols.
The layout of a data structure changed. I'm not sure how to detect that
from readelf/objdump output.

>> Maybe we can replace it with a hash of all installed header files. This
>> might be a better way to resolve cases like this wolfssl instance, but
>> we likely will still be rebuilding dependent packages on most updates.
> 
> A hash of installed header files makes little sense imho as it does not really
> tell which symbols are exported by a shared object in the end. To me it seems
> you conflate two different problems here:
> 
> a) maintaining ABI versions
> b) ABI depending on compile time optionsI'm not conflating the two. I'm saying that in the instance that I ran
into, the compile time options were the cause, but the last package
version update (which did not increase ABI_VERSION or soname) introduced
ABI changes that could have very similar effects.
Simple runtime tests often don't catch these issues (in my test only a
very specific runtime config ran into it), soname version changes are
not a reliable indicator, symbol changes aren't enough either.

How do we address this in a proper way that isn't a potential foot-gun
on every version upgrade?

- Felix