[PATCH] scripts: add size_compare.sh
Paul Spooren
mail at aparcar.org
Mon Nov 16 14:32:07 EST 2020
On Thu Sep 3, 2020 at 3:36 AM HST, Adrian Schmutzler wrote:
> > -----Original Message-----
> > From: openwrt-devel [mailto:openwrt-devel-bounces at lists.openwrt.org]
> > On Behalf Of Paul Spooren
> > Sent: Freitag, 31. Juli 2020 05:15
> > To: openwrt-devel at lists.openwrt.org
> > Cc: freifunk at adrianschmutzler.de; Paul Spooren <mail at aparcar.org>
> > Subject: [PATCH] scripts: add size_compare.sh
> >
> > As package size changes are a continuous topic on the mailing list this scripts
> > helps developers to compare their local package modifications against latest
> > upstream.
> >
> > The script downloads the latest package indexes based on env variables or
> > the `.config` file. The script compares the actual installed size
> > (data.tar.gz) or the IPK package size.
>
> Just gave this a quick try, and generally it looks good.
>
> However, I got 82 packages listed for " Compare packages of
> ath79/generic/mips_24kc", but only 66 were in the final comparison.
The current script also interprets some options as packages, which is
couldn't find a clean solution for:
CONFIG_PACKAGE_MAC80211_DEBUGFS=y
# CONFIG_PACKAGE_MAC80211_TRACING is not set
CONFIG_PACKAGE_MAC80211_MESH=y
Also, this script only checks for packages of openwrt.git, so if you
have packages from other repositories installed, they'd be missing.
>
> Will check the code in detail later. A few comment below.
>
> >
> > An example output is found below:
> >
> > ```
> > user at dawn:~/src/openwrt/openwrt$ ./scripts/size_compare.sh Compare
> > packages of ath79/tiny/mips_24kc:
> > dropbear busybox iw ubus
> >
> > Checking installed size
> >
> > Fetching latest package indexes...
> > % Total % Received % Xferd Average Speed Time Time Time Current
> > Dload Upload Total Spent Left Speed
> > 100 80634 100 80634 0 0 33499 0 0:00:02 0:00:02 --:--:-- 33485
> > % Total % Received % Xferd Average Speed Time Time Time Current
> > Dload Upload Total Spent Left Speed
> > 100 54082 100 54082 0 0 24252 0 0:00:02 0:00:02 --:--:-- 24252
> >
> > Comparing package sizes...
> > Change Local Package
> > 611B 208715B busybox
> > 39B 5612B ubus
> > -42B 34940B iw
> > -14916B 89265B dropbear
> > ```
> >
> > I plan to integrate this script into the CI so we have a summary how sizes
> > change over different architectures.
> >
> > Signed-off-by: Paul Spooren <mail at aparcar.org>
> > ---
> > scripts/size_compare.sh | 117
> > ++++++++++++++++++++++++++++++++++++++++
> > 1 file changed, 117 insertions(+)
> > create mode 100755 scripts/size_compare.sh
> >
> > diff --git a/scripts/size_compare.sh b/scripts/size_compare.sh new file mode
> > 100755 index 0000000000..b310a085a4
> > --- /dev/null
> > +++ b/scripts/size_compare.sh
> > @@ -0,0 +1,117 @@
> > +# SPDX-License-Identifier: GPL-2.0-or-later # # Copyright (C) 2020 Paul
> > +Spooren <mail at aparcar.org> # ### ### size_compare - compare size of
> > +OpenWrt packages against upstream ### ### The script compares locally
> > +compiled package with the package indexes ### available upstream. This
> > +way the storage impact of optimizations or ### feature modifiactions is
> > +easy to see.
> > +###
> > +### If no environmental variables are set the scritp reads the current
> > +### .config file. The evaluated env variables are the following:
> > +###
> > +### TARGET SUBTARGET ARCH PACKAGES BIN_DIR BASE_URL
> > CHECK_INSTALLED
> > +###
> > +### Usage:
> > +### ./scripts/size_compare.sh
> > +###
> > +### Options:
> > +### -p --package-size Check IPK package size and not installed size
> > +### -h --help This message
> > +
> > +CONFIG_TARGET=$(sed -n 's/^CONFIG_TARGET_BOARD="\(.*\)"$/\1/p'
> > .config)
> > +CONFIG_SUBTARGET=$(sed -n
> > 's/^CONFIG_TARGET_SUBTARGET="\(.*\)"$/\1/p'
> > +.config) CONFIG_ARCH=$(sed -n
> > +'s/^CONFIG_TARGET_ARCH_PACKAGES="\(.*\)"$/\1/p' .config)
> > +CONFIG_PACKAGES=$(sed -n 's/^CONFIG_PACKAGE_\(.*\)=y$/\1/p'
> > .config |
> > +tr '\n' ' ') CONFIG_BIN_DIR=$(sed -n
> > +'s/^CONFIG_BINARY_DIR="\(.*\)"$/\1/p' .config)
> > +
> > +TARGET=${TARGET:-$CONFIG_TARGET}
> > +SUBTARGET=${SUBTARGET:-$CONFIG_SUBTARGET}
> > +ARCH=${ARCH:-$CONFIG_ARCH}
> > +PACKAGES=${PACKAGES:-$CONFIG_PACKAGES}
> > +BIN_DIR=${CONFIG_BIN_DIR:-./bin}
> > +BASE_URL="${BASE_URL:-https://downloads.openwrt.org/snapshots}"
> > +CHECK_INSTALLED="${CHECK_INSTALLED:-y}"
> > +
> > +TARGET_URL="$BASE_URL/targets/$TARGET/$SUBTARGET/packages/Pack
> > ages.gz"
> > +PACKAGES_URL="$BASE_URL/packages/$ARCH/base/Packages.gz"
> > +
> > +help() {
> > + sed -rn 's/^### ?//;T;p' "$0"
> > +}
> > +
> > +package_size () {
> > + FOUND_PACKAGE=
> > + if [ -z "$CHECK_INSTALLED" ]; then
> > + SEARCH_PATTERN="Size"
> > + else
> > + SEARCH_PATTERN="Installed-Size"
> > + fi
> > + while IFS= read -r line; do
> > + if [ "$line" = "Package: $2" ]; then
> > + FOUND_PACKAGE=y
> > + fi
> > + if [ -n "$FOUND_PACKAGE" ]; then
> > + case $line in
> > + "$SEARCH_PATTERN"*)
> > + echo "$line" | cut -d ' ' -f 2
> > + break
> > + ;;
> > + esac
> > + fi
> > + done < "$1"
> > +}
> > +
> > +compare_sizes () {
> > + for PACKAGE in $PACKAGES; do
> > + if [ "$PACKAGE" = "libc" ]; then
> > + continue
> > + fi
> > + PACKAGE_FILE=$(find "$BIN_DIR/packages/$ARCH/"
> > "$BIN_DIR/targets/$TARGET/$SUBTARGET/" -name "${PACKAGE}_*.ipk")
>
> When I first run this, I was in x86 target where only a kernel refresh
> has happened before (so, no build).
> This then produces a lot of warning due to missing packages folder:
>
> find: ‘./bin/targets/x86/generic/’: No such file or directory
> find: ‘./bin/packages/i386_pentium4/’: No such file or directory
>
> So, one should either suppress the message here for the find command, or
> alternatively check for the folders' existence beforehand.
I surpressed the error message.
>
> > + if [ -z "$PACKAGE_FILE" ]; then
> > + continue
> > + fi
> > + if [ -z "$CHECK_INSTALLED" ]; then
> > + SIZE_LOCAL=$(stat -c '%s' "$PACKAGE_FILE")
> > + else
> > + SIZE_LOCAL=$(tar tzvf "$PACKAGE_FILE" | grep
> > data.tar.gz | awk '{ print $3 }')
> > + fi
> > + SIZE_UPSTREAM=$(package_size "$TMP_INDEX"
> > "$PACKAGE")
> > + SIZE_DIFF="$((SIZE_LOCAL-SIZE_UPSTREAM))"
> > + echo "${SIZE_DIFF}B ${SIZE_LOCAL}B $PACKAGE"
>
> Personally, I think the "B" looks ugly. As we only have bytes, we could
> consider to drop it.
Dropped.
> One could also consider to add a "+" for the first column when change is
> positive. Both changes here are lowest priority.
Added.
I created a updated version on GitHub as I'm not the biggest fan of
patchworks: https://github.com/openwrt/openwrt/pull/3599
> > + done
> > +}
> > +
> > +if [ "$1" = "-h" ]; then
> > + help
> > + exit 0
> > +fi
> > +
> > +if [ "$1" = "-p" ]; then
> > + CHECK_INSTALLED=
> > +fi
> > +
> > +echo "Compare packages of $TARGET/$SUBTARGET/$ARCH":
> > +echo "$PACKAGES"
> > +echo
> > +
> > +if [ -z "$CHECK_INSTALLED" ]; then
> > + echo "Checking IPK package size"
> > +else
> > + echo "Checking installed size"
> > +fi
> > +echo
> > +
> > +echo "Fetching latest package indexes..."
> > +TMP_INDEX=$(mktemp /tmp/size_compare_package_index.XXXXXX)
> > +curl "$TARGET_URL" | gzip -d > "$TMP_INDEX" || exit 1 curl
> > +"$PACKAGES_URL" | gzip -d >> "$TMP_INDEX" || exit 1 echo
> > +
> > +echo "Comparing package sizes..."
> > +echo "Change Local Package"
>
> Since we have enough space, I'd go for "Change Local Remote Package"
> here.
>
> Best
>
> Adrian
>
> > +compare_sizes | sort -n -r
> > +
> > +rm "$TMP_INDEX"
> > --
> > 2.25.1
> >
> >
> > _______________________________________________
> > openwrt-devel mailing list
> > openwrt-devel at lists.openwrt.org
> > https://lists.openwrt.org/mailman/listinfo/openwrt-devel
More information about the openwrt-devel
mailing list