[OpenWrt-Devel] "UCI overlay" proposal

[Zefir Kurtisi]

Hello Matthias,

can't provide qualified feedback to your proposal, but at least want to post my
experience and hope that it is valuable for the discussion.

Our company develops highly specialized WLAN products for decades, for a long time
based on OpenWRT. The niche market we are in has a strict requirement for
long-term availability and compatibility. In the context of configuration, in
addition to the default Web-UI configurability, we must support SNMP and stay
compatible with MIBs we initially released like 10 years ago.

Consequently, for a long time we are following an approach similar to what you are
proposing. In the beginning, the config overlay was a simple mapper between
backend (uci) and frontend (snmp). Later, when config space and complexity grew,
we defined and used a 'meta-config-space' that was generated at runtime from uci
to simplify the mapping to snmp.

Alas, what looked like a reasonable and clever approach, over time turned into
maintenance-hell. We were faced with two moving targets (uci config gradually
changing, MIB extensions) that need continuous synchronization and render the
approach error-prone and resource-demanding. From that experience, I tend to
believe that an overlay- or meta-configuration is only practicable as interim
solution during a transition towards a standardized config space.

If I had a chance to go back and do things better, this would be my approach:

1) define standardized config space
The hardest of all parts - config space must be generic and complete enough to
cover each and every potential setup, while at the same time it has to be
extensible, backwards-compatible, and well defined. Whoever is familiar with
standardization processes knows this will be no picnic. On the good side, with
ubus gradually becoming the core framework for OpenWRT, the format for the new
config to be JSON is more or less given. As for the structure, in [1] an
announcement for a generic network configuration standard in JSON was posted that
might well serve as starting point.
2) implement interim overlay
Once the config space in 1) is standardized, implement an temporary overlay that
provides transparent access to the configuration via legacy uci and new JSON
config spaces. From that point on, new applications / packages must not use uci
any more.
3) gradually move to new config space
Over a defined (and relatively short) transition period, existing packages has to
be converted to use the JSON config. At the end of that period, uci should not be
used any more.


I have developed a few private tools for the overlay-config approach (like
conversion from uci to JSON), which are not suitable for upstreaming (C++,
proprietary dependencies). From there, my estimate for the implementation workload
for the transition (after the config space was standardized) is around 3 man-months.


Good Luck,
Zefir


[1] https://lists.openwrt.org/pipermail/openwrt-devel/2015-January/030422.html


On 03/05/2015 01:36 PM, Matthias Schiffer wrote:
> Hi,
> during the development of our Freifunk firmware framework "Gluon" we've
> come to the conclusion that the current UCI configuration format using
> static files doesn't always fit our needs. Therefore we propose a new
> feature we've called "UCI overlays".
> 
> The basic idea is that we'd like to provide UCI configuration by other
> means than the static files in /etc/config, for example from a database
> or generated by scripts on the fly (the latter being our main usecase -
> we'd like to generate configuration for netifd and fw3 based on
> meta-configuration data). This should work transparently, without
> needing changes in the config consumers (applications).
> 
> The overlay-provided configuration packages should be merged with the
> config read from /etc/config. We'd like to generate both config sections
> which may be overruled by corresponding options in /etc/config, and
> read-only sections which can't be changed by the user through UCI.
> 
> We see two different ways to implement this:
> 
> (1) Make the "overlay" an alternative backend which uses the "file"
> backend to merge generated config with the one from files
> 
> (2) Introduce overlays as a new concept in libuci
> 
> 
> While the first option would need less changes in libuci, it doesn't
> allow stacking different overlays, so we're in favour of option 2.
> 
> Both ways would need a config file (/etc/uci.conf?) to configure the
> overlays being used. Our plan is to implement overlays as dlopen-able
> plugins loaded from somewhere like /lib/uci/overlay so it is possible
> for other packages to provide overlay implementations.
> 
> In addition, we'd like to add a new attribute "readonly" to the
> uci_element structure so changing read-only configuration will fail as
> soon as someone tries to uci_set it.
> 
> Does this sound reasonable? We can develop the needed extensions
> ourselves, but of course we'd like to get this feature upstream to avoid
> carrying the patches downstream indefinitly, so we're eager to know what
> you think of this proposal.
> 
> Regards,
> Matthias
> 
> 
> 
> _______________________________________________
> openwrt-devel mailing list
> openwrt-devel at lists.openwrt.org
> https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel
> 
_______________________________________________
openwrt-devel mailing list
openwrt-devel at lists.openwrt.org
https://lists.openwrt.org/cgi-bin/mailman/listinfo/openwrt-devel