SquashFS mixed errors (decompression failed and others)

[Vincent Wiemann]

Hi Ibrahim,

please post the affected model name. It could e.g. be a bad board design
which requires the flash to be clocked slower or something.
Without knowing the affected board, it is guessing...

Best,

Vincent

On 5/21/21 1:19 PM, Ibrahim Tachijian wrote:
> Hello,
> 
> We use approximately 10k IPQ40XX devices and we have noticed that
> every time we run "sysupgrade -n" we lose approximately 1% of the
> routers in the process.
> After further investigation I'm almost confident that it is not the
> sysupgrade process that is the culprit - so what I did was that I put
> one test router into a reboot loop.
> 
> This is what I do;
> 
> Boot the router in a fresh state after a newly installed image.
> The image contains a reboot loop that consists of a shell script that
> runs every minute.
> 
> The shell script tries to run a php-script which simply echoes "Hello
> World". If the php-script exists normally then we reboot the router.
> 
> However the php-script exists abnormally then the router stops and
> does nothing other than informing me that there was a bus-error making
> php not able to process the hello world script.
> 
> When this process runs the router reboots approximately 50 times
> before it boots into a state which is faulty where I see bus-errors
> when I try to run php scripts for example.
> 
> 
> Looking into dmesg you can see some errors such as,
> 
> [10985.209438] SQUASHFS error: squashfs_read_data failed to read block 0x3a803e
> [11045.218685] SQUASHFS error: xz decompression failed, data probably corrupt
> [11045.218731] SQUASHFS error: squashfs_read_data failed to read block 0x3a803e
> [11105.228157] SQUASHFS error: xz decompression failed, data probably corrupt
> [11105.228203] SQUASHFS error: squashfs_read_data failed to read block 0x3a803e
> 
> or
> 
> [26218.687905] SQUASHFS error: Unable to read page, block 1b99a, size 10234
> [26221.057472] SQUASHFS error: Unable to read data cache entry [1b99a]
> [26221.057551] SQUASHFS error: Unable to read page, block 1b99a, size 10234
> [26221.062926] SQUASHFS error: Unable to read data cache entry [1b99a]
> [26221.069742] SQUASHFS error: Unable to read page, block 1b99a, size 10234
> [26224.460239] SQUASHFS error: Unable to read data cache entry [1b99a]
> [26224.460320] SQUASHFS error: Unable to read page, block 1b99a, size 10234
> 
> or
> 
> [62745.801178] SQUASHFS error: squashfs_read_data failed to read block 0x732ae2
> [62773.347234] SQUASHFS error: xz decompression failed, data probably corrupt
> [62773.347281] SQUASHFS error: squashfs_read_data failed to read block 0x732ae2
> [62790.132661] SQUASHFS error: xz decompression failed, data probably corrupt
> [62790.132706] SQUASHFS error: squashfs_read_data failed to read block 0x732ae2
> [62790.216746] SQUASHFS error: xz decompression failed, data probably corrupt
> [62790.216792] SQUASHFS error: squashfs_read_data failed to read block 0x732ae2
> [62800.810525] SQUASHFS error: xz decompression failed, data probably corrupt
> [62800.810570] SQUASHFS error: squashfs_read_data failed to read block 0x732ae2
> [62828.336267] SQUASHFS error: xz decompression failed, data probably corrupt
> 
> 
> 
> Now, you would assume that the squashfs-partition is broken - but if
> this was the case then a reboot should not help. It does.
> Rebooting the router after it boots in this faulty state fixes the issue.
> 
> So approximately 1-2% of my reboots make the router go into this faulty state.
> 
> I am clueless on how to further investigate this issue. For now my
> work around is restarting the router via a bash script should it
> notice there are bus-errors or i/o errors.
> 
> Thanks
> 
>