[OpenWrt-Devel] "UCI overlay" proposal

[Matthias Schiffer]

On 03/05/2015 08:25 PM, David Lang wrote:
> On Thu, 05 Mar 2015 13:36:10 +0100, Matthias Schiffer wrote:
>> Hi,
>> during the development of our Freifunk firmware framework "Gluon" we've
>> come to the conclusion that the current UCI configuration format using
>> static files doesn't always fit our needs. Therefore we propose a new
>> feature we've called "UCI overlays".
>>
>> The basic idea is that we'd like to provide UCI configuration by other
>> means than the static files in /etc/config, for example from a database
>> or generated by scripts on the fly (the latter being our main usecase -
>> we'd like to generate configuration for netifd and fw3 based on
>> meta-configuration data). This should work transparently, without
>> needing changes in the config consumers (applications).
>>
>> The overlay-provided configuration packages should be merged with the
>> config read from /etc/config. We'd like to generate both config sections
>> which may be overruled by corresponding options in /etc/config, and
>> read-only sections which can't be changed by the user through UCI.
>>
>> We see two different ways to implement this:
>>
>> (1) Make the "overlay" an alternative backend which uses the "file"
>> backend to merge generated config with the one from files
>>
>> (2) Introduce overlays as a new concept in libuci
>>
>>
>> While the first option would need less changes in libuci, it doesn't
>> allow stacking different overlays, so we're in favour of option 2.
>>
>> Both ways would need a config file (/etc/uci.conf?) to configure the
>> overlays being used. Our plan is to implement overlays as dlopen-able
>> plugins loaded from somewhere like /lib/uci/overlay so it is possible
>> for other packages to provide overlay implementations.
>>
>> In addition, we'd like to add a new attribute "readonly" to the
>> uci_element structure so changing read-only configuration will fail as
>> soon as someone tries to uci_set it.
>>
>> Does this sound reasonable? We can develop the needed extensions
>> ourselves, but of course we'd like to get this feature upstream to avoid
>> carrying the patches downstream indefinitly, so we're eager to know what
>> you think of this proposal.
> 
> Should this be built in to uci, or should this be a set of utilities
> that take /etc/config/wireless.d/* and create /etc/config/wireless when
> run (which then gets used normally)?
> 
> You can then have scripts, database tools, config management tools
> (puppet, chef, etc) drop things in such directories and the utilities
> can run in cron (or use inotify in an advanced setup) to notice that
> there was a change and do the appropriate thing to re-read the config.
> 
> Building it into uci is more elegant, but I worry that having it that
> integrated will mean that each new way to create config info will end up
> requiring changes to uci. File based is a little less elegant, but it
> makes it much easier for people to add new ways to do things.
> 
> If you are assembling /etc/config/wireless from /etc/config/wireless.d/*
> you can just say that the resulting file is the combination of all the
> included files, and it's up to the admin to avoid conflicts between the
> pieces. If you take /etc/config/wireless and overlay other things on it,
> you have the question of how do you negate something that was in the
> base file.
> 
> 
> 
> I do think that something along these lines should be done. It would be
> very nice to be able to break up existing config files a little bit (for
> example, /etc/config/network split raw interfaces from bridge configs
> from switch configs). And it would be great to be able to have a config
> management tool be able to just add-on to the base config, replacing
> part of it, without the config management tool needing to manage
> everything in the file.
> 
> Over on the CeroWRT list we are discussing what things are in CeroWRT
> that have not yet been unstreamed into OpenWRT, and most of them come
> down to config items. Some of them are 'fairly' small (like renaming
> interfaces by function, so you have an interface named WAN rather than
> eth0), while others are very fundamental (routing between each wifi
> interface and the wired network instead of bridging). An overlay or
> name.d approach would make it much easier for these different approaches
> to be presented as packages that can be added to OpenWRT without forcing
> everyone to change to them.
> 
> David Lang


At the moment we use scripts to edit /etc/config/* to our needs. There
are multiple reasons why we'd like to get rid of this:

* Changing specific parts of meta-configuration may change large parts
of network and firewall configuration. We'd like to avoid writing to the
flash more than necessary.
* As long as the UCI config is written to files, admins think they can
change them, not anticipating that parts of the config may be rewritten,
leading to all kinds of funny misconfigurations. Not actually having the
config written out, but generated on the fly, is also an attempt to make
the configuration more "fool-proof", by leaving the meta-config as the
only way to make changes to the configuraion (in the case of read-only
overlays)

I can understand your concerns, I'd also like to avoid having to change
UCI more than necessary (and more often than necessary). The minimal
solution (which is the option 1 I'd proposed) would be just adding
support for dlopen-based backend loading and making the default backend
configurable (in /etc/uci.conf or something like that).

Option 2 would be nicer in my opinion. I can see other usecases besides
ours which would benefit from a way to combine configuration from
different sources, so having a generic way to do that in the UCI core
could facilitate maintaining such solutions.

I've also though about just calling this "multi-backend support" instead
of "overlays". While that would be very nice and generic for the config
consumer, not having a single actual backend would complicate the config
write case a lot...

Thanks for your input, I hope we can find a solution which benefits both
our projects.

Regards,
Matthias

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 819 bytes
Desc: OpenPGP digital signature
URL: <http://lists.infradead.org/pipermail/openwrt-devel/attachments/20150307/8c2d0ec2/attachment.sig>
-------------- next part --------------
_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel