Buildbot stuck? No new packages builds since Tuesday 22nd

[Hauke Mehrtens]

On 6/27/21 9:25 AM, Hannu Nyman wrote:
> Hi,
> Looks like the workers have got stuck again. All fsf-dock workers. Only 
> the two different workers that were just started are still alive.
> 
> The same seems to concern also the 21.02 packages buildbot. All fsf-dock 
> workers seem stuck.

Hi Hannu,

This is a different problem. The Oregon State University Open Source Lab 
has to power down all open compute systems due to an excessive heat warning:
https://forecast.weather.gov/showsigwx.php?warnzone=ORZ008&warncounty=ORC003&firewxzone=ORZ603&local_place1=Corvallis%20OR&product1=Excessive+Heat+Warning&lat=44.5647&lon=-123.261

We were informed about this yesterday and they hope they can start them 
again today or tomorrow.

> Ps. the 21.02 buildbot looks severely under-resourced compared to the 
> half-dormant 19.07 branch. I would think that the current RC release 
> should already have the priority for build resources.

It is probably a good idea to switch some of the resources there are 
also much less commits in the 19.07 branch.

Hauke

> On 2021-06-26 13:59, Hauke Mehrtens wrote:
>> On 6/26/21 10:15 AM, Hannu Nyman wrote:
>>> Looks like the packages buildbot is somehow stuck. No new builds has 
>>> been uploaded since Tuesday 22nd.
>>> https://downloads.openwrt.org/snapshots/packages/
>>>
>>> Buildbot status shows only janitor activity for the last four days.
>>> https://buildbot.openwrt.org/master/packages/#/waterfall
>>
>> Hi Hannu,
>>
>> Thanks for reporting this problem, Baptiste fixed it by restarting the
>> build bots.
>>
>> Hauke
>>
>>
>> zorun | there are connection failures in the log on 22/06, and then
>>         regulary since then: [Broker (TLSMemoryBIOProtocol),client]
>>         Master replied to keepalive, everything's fine
>> zorun | so the workers are connected to the master and are waiting for a
>>         job, but the master doesn't send anything
>> zorun | it looks like a synchronisation bug in buildbot :/
>> zorun | ynezz: if you want to have a look, I have left fsf-dock-24 in
>>         this "stuck" state
>> zorun | I have restarted all other workers and they are now building

-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_0x93DD20630910B515.asc
Type: application/pgp-keys
Size: 9895 bytes
Desc: OpenPGP public key
URL: <http://lists.openwrt.org/pipermail/openwrt-devel/attachments/20210627/22552fe8/attachment-0001.bin>
-------------- next part --------------
A non-text attachment was scrubbed...
Name: OpenPGP_signature
Type: application/pgp-signature
Size: 488 bytes
Desc: OpenPGP digital signature
URL: <http://lists.openwrt.org/pipermail/openwrt-devel/attachments/20210627/22552fe8/attachment-0001.sig>