[OpenWrt-Devel] [OpenWrt] openwrt build system costs: support of or foundation over?

Tue Dec 8 11:39:10 EST 2015

On Mon, Dec 7, 2015 at 8:26 PM, Eric Schultz
<eschultz at prplfoundation.org> wrote:
> Dave,
>
> Thanks for bringing this up. Based upon my conversations at the OpenWrt
> summit and observations, an official OpenWrt Foundation would be a positive
> development. I heard at least a few companies say that they want to help but
> they don't even know who to pay in some cases.

I was figuring that nobody knows who to pay in most cases!

This was why my thinking was more towards having a broker-like
arrangement for various things. We put up a spreadsheet of existing
costs. it's pretty easy to defend expanding the build cluster
system(s) to some point of diminishing return, and merely getting that
tiny amount done - with X number of companies contributing to it -
would be a start. Everybody can then point proudly at everything more
often lit up green at:

http://buildbot.openwrt.org:8010/grid

on their "what I'm doing for open source" promo material - and revel
in a personal or product build that works the first time more often -
and everybody's happy.....

Having a set of goals clearly expressed by all sides on this, and
other matters, openly, would be a start.

What is happening instead: companies that should be investing a little
in basic infrastructure, into basic tools, and - building quality,
responsiveness, and good software engineering methods into everything
they use...

instead get focused on features that they want, that's "missing", and
the quality of the rest of the system is hit or miss.

I was struck a few days back looking at:

http://rdkcentral.com/about-rdk/

when 99.99% of the code, and quality, needs to be delivered in the
linux portion of the stack, which is otherwise represented as this
tiny, "obviously working" slice of the chart. They are still running
Linux 2.6 over there last I looked....

progress and quality does not happen automagically. There has always
been a disconnect between engineers building better software, and
companies shipping products, and it would be nice to find more ways to
communicate across and close the gaps.

This is one of my all time favorite satires of the problems all sides
face in delivering good products:

https://www.youtube.com/watch?v=KowAEqHpftI

> There's lots of reasons why
> this has been the case but, in the end, more clarity in governance and
> process would go a long way to help here. I'm happy to personally help but
> the committers are really the ones who have to drive this.

And they are pretty busy and tend to disagree a lot. :) Which was part
of openwrt's strength during the Great recession - they couldn't be
bought and folded up like embedded alley and montavista were.

Something - whatever that thing is - needs to be democratic, neutral,
and trusted. Perhaps writing down everybody wishlist of needs would be
a start...

> On a related topic, prpl is contracting with me to facilitate and help build
> a distributed test platform for OpenWrt. The goal is somewhat similar in
> design to Kernel CI but handles testing of upstream nightlies on devices as
> well as boot. I'd like to have you involved and I'll be sending out the
> participation information soon for next week's  kickoff meeting. Obviously,
> that platform will require regularly built images so if those don't exist
> then we would have to address that.

Good. We need more and better testbeds, and I certainly agree that
getting to where stuff could be flashed, abused, and tested on a
regular basis would be great. I really hate it when I flash a brick.

testbed note here:

Over in make-wifi-fast land at kau.se we finally are doing testing of
the cake algorithms on a realistic testbed that can simulate long rtts
to 105mbit. It took 6 months to get to this point....

And we are starting to boot up the ath9k based x86 testbed where we
hope to at least get to gbit, on the budget we had. It would be nice
to actually drive the testbed with 10GigE so as to get to at least
1.5Gbit (802.11ac) but oy, we costed that out and we're not going to
be able to do it.

So if there is a way for bufferbloat.net, my (or other interested unis
or orgs)  to apply to prpl for a grant to support the make-wifi-fast
project, it would be nice. prpl does not seem to have a public grant
process....

> I personally didn't realize there were
> significant funding issues with the build infrastructure;

I see a few more servers have come online since last I checked.

Define "significant"? 220/mo is 1/10th my current base salary after tax.

> I know prpl in the
> past has offered to pay for the OpenWrt hosted build server but that offer
> wasn't accepted.

Oy. Nobody asked me... I'll send the last year's - or this month's,
worth of build cluster invoices - to anyone willing to foot the bill.

A more general question to ask is: to just, like, ask the list(s), is:
"my foundation has $XX,000/yr to contribute to making openwrt better.
how would y'all like to spend it? ".

"We also have one time grants available for X and Y amounts - How have
you been supporting openwrt, how can we thank you?"

> If you and the OpenWrt team could explain the situation
> better, what's hosted where, what the costs are and I understand the
> situation better, I can recommend to prpl leadership that funding be
> considered.

My take on it was about 1.7k/month of google compute got all basic
builds done for everything in under 12 hours each. Arguably more than
one cluster should be used (security/safety/redundancy) and more than
one linux distro used to build it, and if there were ways to get the
build time down, or just test build new stuff faster, great...

> Eric
>
> On Mon, Dec 7, 2015 at 12:51 PM, Dave Taht <dave.taht at gmail.com> wrote:
>>
>> I am still not sure what prpl is for. Nor, what happens to donated
>> funds, if any...
>>
>> I have long wanted some non-profit org to serve as an intermediary
>> between the profit centered corps and the developers. I don't think
>> SPI or prpl is doing this right. Linaro sort of used to...
>>
>> There needs to be some sort of translation between the three parties
>> of real needs. A corp is used to paying X for Y, in particular, not
>> fuzzy donations...
>>
>> As sort of a test case for what I'm trying to explain, there are a few
>> people that contribute towards keeping the Continuous Integration and
>> build system of openwrt up and running, both in terms of hardware, and
>> time.
>>
>> Many have been doing it longer than I have, but for most of the last 4
>> years a lot of the visible hardware came from me (snapon, huchra,
>> gb-XX).
>>
>> For 14 months of that, courtesy of a google cloud grant that expired
>> last year, I was running 6(?) boxes at a cost of about $1750/month, +
>> 2 donated boxes from isc.org - and builds popped out in half a day for
>> everything.  I liked it....
>>
>> I had to shut all but one machine off after that grant expired, and
>> isc has (sadly) just shut down their free non-profit hosting service
>> also... huchra is nearly gone, snapon has moved to sweden behind a fw
>> where it can't act as a host...
>>
>> ... and recently I see that the lack of enough machines in the build
>> system was annoying enough to at least one party to start up a new
>> box...
>>
>> ... but - dang it! -  the benefits of CI for openwrt should be
>> *obvious to everyone that uses it*. Especially including some big
>> corps for which the opex of a few thousand dollars a month isn't even
>> noticeable.
>>
>> HP supports debian's build cluster, for example.
>>
>> Now, of late, I've had a devil of a time keeping the lights on. My
>> contribution to the cluster costs me $220/month that I would rather
>> spend on... food, and fixing wifi in general, etc [1]
>>
>> So... I'd like it if there was some org(s) were paying these costs
>> (and for everyone else contributing a box themselves) rather than me.
>> That is a "support of" sort of thing, where control remains in the
>> hands of engineers that care. (In fact, I don't have to care, travis
>> takes care of the problems, I just pay the opex bill)
>>
>> If some org were to "take over" this responsibility, the control slips
>> to that org - gains management - and other BS - and the CI might not
>> get done as well.
>>
>> If some org were to however, take a wishlist of existing costs from
>> existing developers, turn that into a budget, present that to willing
>> commercial orgs, and turn that around to the requesting dev (and
>> publicly), then everyone's lives would be better. X for Y with the
>> help of Z.
>>
>> Mine, and (I think) most of openwrt folk's resistance to
>> "organizations" comes from the top down attempt at exerting control in
>> exchange for money.
>>
>> ...
>>
>> Certainly the build system could get done better! I was very happy
>> seeing benchmarks go by with how much faster they could be done... and
>> doing that right does involve human resources that might want to get
>> paid also... the main reason why gb-10 still exists is because it was
>> weeks of time to get it running in the first place, and easy to
>> replicate, and I'd hate to lose those invested weeks were more grant
>> money to arrive...
>>
>> Now: I have no intention of shutting down gb-10, but I came within
>> hours of having to do so, last month. Got saved by a shuttleworth
>> flash grant...
>>
>> At the moment I am just sending the personal - not a single !@#!
>> corporate - donations I get at the below url to keep it running, and
>> not thinking about it very hard - but I just reached for some spare
>> cash to buy a new board, and came up empty.
>>
>> it's the meta problem, of keeping infrastructure beneath the devs, in
>> general, with a minimal amount of overstructure on top, that's bugging
>> me today.
>>
>> thx for listening. ideas?
>>
>>
>> Dave Täht
>> Keeping the lights on for open routers
>> [1] https://www.patreon.com/dtaht?ty=h
>> _______________________________________________
>> OpenWrt mailing list
>> OpenWrt at lists.prplfoundation.org
>> http://lists.prplfoundation.org/cgi-bin/mailman/listinfo/openwrt
>
>
>
>
> --
> Eric Schultz, Community Manager, prpl Foundation
> http://www.prplfoundation.org
> eschultz at prplfoundation.org
> cell: 920-539-0404
> skype: ericschultzwi
> @EricPrpl
_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel