[OpenWrt-Devel] [PATCH v2 0/3] build: Avoid 'Argument list too long' error

[Eneas U de Queiroz]

TLDR for those who have read v1:  I've come up with a good-enough, but
not perfect wildcard pattern, and added it in a separate commit.

After building all of the luci packages with all of the translations,
if I either run:
'make package/luci/clean' or 'make package/luci/compile', I get:
make[2]: execvp: /usr/bin/env: Argument list too long
make[2]: *** [../../luci.mk:285: luci-clean] Error 127

This is caused by the call to scripts/ipkg-remove with a list of over
2,300 packages matching a luci* wildcard:
$ ll bin/packages/arm_cortex-a9_vfpv3/luci/luci*.ipk | wc -l
2307

My first attempt to circumvent this was using xargs.  However, using
echo from the Makefile results in make calling the shell with the same
2,307 file names.  The solution is to have make write the list to a file
and then feed it to xargs.

To avoid creating a file every time, xargs is only used when the number
of files is >=512.

As an optimization, to avoid calling wildcard twice, I've defined a
RemoveOpkgPackageFiles function, and added the check for an empty list
there, so that the call to opkg_package_files would only be done when
the new function was called.

I've put them in separate commits to ease an eventual reversal or
rejection.

If we change the wildcard pattern that selects the files, we can
eliminate the problems with xargs, and avoid 4,612 runs of 'tar -Ozxf'
when making package/luci/compile.

There is a caveat; it will not remove the .ipk file if the version of
a package whose name ends in a digit (e.g. nghttp2) that was currently
built with an ABI_VERSION, but the version of the new build does not
have an ABI_VERSION.  Then, make package/nghttp2/clean will not remove
the old ipk file.

I consider this extremely minor.  Nonetheless, I will leave the
intermediate commit, since it works in 100% of the cases, making it
easier to revert this.  If I should rather squash them, let me know
which commits should be kept and I'll send a v3.

This was compile-tested for mvebu, and checked by adding $(info ...)
tracers to the new functions and to opkg_package_files in
include/feeds.mk.

To check how many ipk files each package was selecting with the new
wildcard, I've run the following script in a directory containing all
packages that I've build for mvebu:

#!/bin/bash
total=0
packages=0
for f in *.ipk; do
  PKG=$(tar -Ozxf "$f" ./control.tar.gz | tar -Ozxf - ./control \
        | sed -ne '/^Package:/{s,.*: ,,; p}')
  SRC=$(tar -Ozxf "$f" ./control.tar.gz | tar -Ozxf - ./control \
        | sed -ne '/^SourceName:/{s,.*: ,,; p}')
  if [ "${SRC}" = "${PKG}" ]; then
    files="${PKG}[^a-z-]*_*.ipk"
  else
    case "${SRC}" in
       *[0-9] )
          files="${SRC}*_*.ipk"
          ;;
       * )
          files="${SRC}[^a-z-]*_*.ipk"
    esac
  fi
  n=$(echo ${files} | wc -w)
  if [ "$n" -ne 1 ]; then
    echo pkg=${SRC} - n=${n}
    ls -1 ${files}
  fi
  ((total = total + n))
  ((packages++))
done
echo Total Packages=${packages}. Total lookups=${total}

If you want to see the old wildcard, always use `files=${SRC}*_*.ipk`.

Before the change:
Total Packages=8213. Total lookups=16689
After:
Total Packages=8213. Total lookups=8838

ChangeLog:

v1->v2:
  * Renamed the new functions using lowercase and underscores
  * Used '< file' instead of 'cat file |' to pass the files to xargs
  * Added a commit changing the wildcard pattern, reverting the use of
    xargs.

Eneas U de Queiroz (3):
  build: package-ipkg: avoid calling wildcard twice
  build: call ipkg-remove using xargs if #args>=512
  build: reduce number of files passed to ipk-remove

 include/package-ipkg.mk | 17 +++++++++++++++--
 1 file changed, 15 insertions(+), 2 deletions(-)


_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/mailman/listinfo/openwrt-devel