[OpenWrt-Devel] "UCI overlay" proposal

[David Lang]

One other thought. If there is a desire to change the format, let's pick a 
format that's already in use for configuring system rather than inventing yet 
another one. We should survey the tools that are being written for managing 
software defined datacenters (openstack and similar) and see if we can use one 
of those config languages (either as-is or with an automated conversion back and 
forth)

David Lang

On Fri, 6 Mar 2015, David Lang wrote:

> changing the format doesn't solve the problem, it just causes churn in all 
> the software.
>
> The problem is the lack of standardization and the fact that the way things 
> are configured changes over time. Not everything is configured in UCI (and 
> this will always be the case because it's not worth trying to change every 
> piece of software available)
>
> When you try to design a "standardized config space" that will work in the 
> future for equipment and software that nobody has imagined yet, you are 
> either going to go crazy-complicated to try and cover everything and have 
> people going cross-eyed trying to understand the spec, or you are going to 
> lock things down to be so ridged that new things will be working around your 
> config spec. The best that you can do is to setup a framework to keep the 
> configs from stepping on each other and try to converge similar software to 
> use the same terms in the config where it makes sense (remember, the newcomer 
> may have a better way of looking at the problem, so trying to force it to 
> define it's config the way the legacy things did it can cause big problems)
>
> I'm all for tools to convert the format to whatever you want to use, and (as 
> I posted earlier), to have tools to assemble/generate the configs, but 
> requiring changes to every piece of software to just change the format 
> requires more than "every langauage can understand JSON"
>
> David Lang
>
> On Fri, 6 Mar 2015, Zefir Kurtisi wrote:
>
>> Hello Matthias,
>> 
>> can't provide qualified feedback to your proposal, but at least want to 
>> post my
>> experience and hope that it is valuable for the discussion.
>> 
>> Our company develops highly specialized WLAN products for decades, for a 
>> long time
>> based on OpenWRT. The niche market we are in has a strict requirement for
>> long-term availability and compatibility. In the context of configuration, 
>> in
>> addition to the default Web-UI configurability, we must support SNMP and 
>> stay
>> compatible with MIBs we initially released like 10 years ago.
>> 
>> Consequently, for a long time we are following an approach similar to what 
>> you are
>> proposing. In the beginning, the config overlay was a simple mapper between
>> backend (uci) and frontend (snmp). Later, when config space and complexity 
>> grew,
>> we defined and used a 'meta-config-space' that was generated at runtime 
>> from uci
>> to simplify the mapping to snmp.
>> 
>> Alas, what looked like a reasonable and clever approach, over time turned 
>> into
>> maintenance-hell. We were faced with two moving targets (uci config 
>> gradually
>> changing, MIB extensions) that need continuous synchronization and render 
>> the
>> approach error-prone and resource-demanding. From that experience, I tend 
>> to
>> believe that an overlay- or meta-configuration is only practicable as 
>> interim
>> solution during a transition towards a standardized config space.
>> 
>> If I had a chance to go back and do things better, this would be my 
>> approach:
>> 
>> 1) define standardized config space
>> The hardest of all parts - config space must be generic and complete enough 
>> to
>> cover each and every potential setup, while at the same time it has to be
>> extensible, backwards-compatible, and well defined. Whoever is familiar 
>> with
>> standardization processes knows this will be no picnic. On the good side, 
>> with
>> ubus gradually becoming the core framework for OpenWRT, the format for the 
>> new
>> config to be JSON is more or less given. As for the structure, in [1] an
>> announcement for a generic network configuration standard in JSON was 
>> posted that
>> might well serve as starting point.
>> 2) implement interim overlay
>> Once the config space in 1) is standardized, implement an temporary overlay 
>> that
>> provides transparent access to the configuration via legacy uci and new 
>> JSON
>> config spaces. From that point on, new applications / packages must not use 
>> uci
>> any more.
>> 3) gradually move to new config space
>> Over a defined (and relatively short) transition period, existing packages 
>> has to
>> be converted to use the JSON config. At the end of that period, uci should 
>> not be
>> used any more.
>> 
>> 
>> I have developed a few private tools for the overlay-config approach (like
>> conversion from uci to JSON), which are not suitable for upstreaming (C++,
>> proprietary dependencies). From there, my estimate for the implementation 
>> workload
>> for the transition (after the config space was standardized) is around 3 
>> man-months.
>> 
>> 
>> Good Luck,
>> Zefir
>> 
>> 
>> [1] 
>> https://lists.openwrt.org/pipermail/openwrt-devel/2015-January/030422.html
>> 
>> 
>> On 03/05/2015 01:36 PM, Matthias Schiffer wrote:
>>> Hi,
>>> during the development of our Freifunk firmware framework "Gluon" we've
>>> come to the conclusion that the current UCI configuration format using
>>> static files doesn't always fit our needs. Therefore we propose a new
>>> feature we've called "UCI overlays".
>>> 
>>> The basic idea is that we'd like to provide UCI configuration by other
>>> means than the static files in /etc/config, for example from a database
>>> or generated by scripts on the fly (the latter being our main usecase -
>>> we'd like to generate configuration for netifd and fw3 based on
>>> meta-configuration data). This should work transparently, without
>>> needing changes in the config consumers (applications).
>>> 
>>> The overlay-provided configuration packages should be merged with the
>>> config read from /etc/config. We'd like to generate both config sections
>>> which may be overruled by corresponding options in /etc/config, and
>>> read-only sections which can't be changed by the user through UCI.
>>> 
>>> We see two different ways to implement this:
>>> 
>>> (1) Make the "overlay" an alternative backend which uses the "file"
>>> backend to merge generated config with the one from files
>>> 
>>> (2) Introduce overlays as a new concept in libuci
>>> 
>>> 
>>> While the first option would need less changes in libuci, it doesn't
>>> allow stacking different overlays, so we're in favour of option 2.
>>> 
>>> Both ways would need a config file (/etc/uci.conf?) to configure the
>>> overlays being used. Our plan is to implement overlays as dlopen-able
>>> plugins loaded from somewhere like /lib/uci/overlay so it is possible
>>> for other packages to provide overlay implementations.
>>> 
>>> In addition, we'd like to add a new attribute "readonly" to the
>>> uci_element structure so changing read-only configuration will fail as
>>> soon as someone tries to uci_set it.
>>> 
>>> Does this sound reasonable? We can develop the needed extensions
>>> ourselves, but of course we'd like to get this feature upstream to avoid
>>> carrying the patches downstream indefinitly, so we're eager to know what
>>> you think of this proposal.
>>> 
>>> Regards,
>>> Matthias
>>> 
>>> 
>>> 
>>> _______________________________________________
>>> openwrt-devel mailing list
>>> openwrt-devel at lists.openwrt.org
>>> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>>> 
>> _______________________________________________
>> openwrt-devel mailing list
>> openwrt-devel at lists.openwrt.org
>> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>> 
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel at lists.openwrt.org
> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
>
_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel