New software configuration and provisioning mechanism

[carlescufi]

Introduction

This issue deals with a topic that has been the center of multiple previous discussions and existing issues and PRs. This is an attempt at briefly summarizing the problem in order to solve it in a way that is agreeable to all potential users and developers alike.

IMPORTANT: Please keep the discussion around this particular feature in this GitHub issue, so that we do not have to collect feedback from multiple sources.

Problem description

Zephyr currently lacks a mechanism to configure multi-instance software components at compile-time well above the hardware. Instead, as described multiple times, we have:

• Devicetree: Multi-instance, but currently restricted to represent (mostly) actual hardware
• Kconfig: Single-instance, mostly restricted to software enablement and configuration

A third mechanism is therefore required. This issue describes the use cases and requirements in order to end up concluding with a final proposal.

Zephyr also lacks a provisioning mechanism, which would be able to generate a pre-compiled "image" of the data that is required in order to provide the required data for certain subsystems to execute.

Use cases

The following use cases have so far been identified:

• Network stack interface description: Handle boot time configuration of network stack #68127
• Configuration of init priorities: [RFC] Init entries: Automatically computing priorities #73836
• Multi-instance software (non-driver) configuration (e.g. buffer sizes in CHAT/CMUX)

We foresee additional users of this mechanism in the future, but the two use cases above exist today and require a solution.

Requirements (WIP)

Requirements document

Proposed diagrams

@nashif:

@fgrandel:

[bjarki-andreasen]

An additional use case is buffer sizes for instances of CHAT (at command handling), CMUX (uart multiplexing), UARTs etc. A CMUX instance can have multiple channels, where only one needs a huge buffer (for network traffic for example) and others require small buffers for typically AT commands.

Initial Use Cases:

• Provisioning a PSA secure storage target (Introduction of a subsystem that implements the PSA Secure Storage API #75275). This needs to be sourced from a secure container at prod time. If we don't want to complicate things at dev and test time then we should integrate this with a simple dev-time config approach for testing.
• I also suggest looking at the use cases I collected from community contributions. Ignoring those would mean ignoring those contributions IMO (none of those comes from me, they've all been collected from previous discussions or asking stakeholders on Discord). Few need to be supported initially but any proposal should be aware of them and include a credible migration path.
• It was mentioned in the arch WG that the sensing subsystem has an immediate stake in the config subsys.

Sources:

• Several people have asked for support of alternative source formats (not now but via a migration path) as input which they generate from internal systems. The requirement here is not that we need multi source at once but it should be clear how we want to migrate there.
• UX for first-time users, this ideally includes a single source for single- and multi-instance configuration.
• A way to cross-reference DT based on a common data model as the entities are mostly the same (required by both, init and net conf)
• A well-defined distinction of Kconfig/DT/sw config that can be reviewed and enforced based on easy-to-observe criteria.
• We need a method to map properties in Kconfig, DT and the new config approach for backwards compatibility as many properties will have to be migrated from both DT and Kconfig over time (as in net config).
• Overlays (required by init right away and probably net conf, too)

Targets:

• type bindings (obvious - just mentioning it for completeness), ideally kept close to usage sites and (as many have suggested) split up based on frequency of updates (maintainability)
• Several people have asked for integration with the settings subsystem (and other NVS). Not required right away, but for net config this is on the migration path.
• Specific configuration properties should be updatable at runtime (net config is an example).
• zero footprint by default (core Zephyr standard for all kinds of config systems)

Documentation:

• Several people have asked for self-documenting bindings, ideally integrated with the doc build system.

[decsny]

my comment copy pasted from the Arch WG chat as requested:

"can we be clear that provisioning requirement is obvious when talking about networking IP address but most multi instance configuration needed to develop software is probably not about provisioning, ie buffer sizes and stack sizes of drivers or properties of how parts of the system will act that doesnt change between serial numbers"

and

"i agree with brix about separating the provisioning problem space from the static multi instance configuration problem space"

[nordicjm]

For network configuration, without having run-time configuration, the whole thing can be done in Kconfig, e.g.

# subsys/Kconfig.network_interfaces file:
# total_interfaces would need to come from some sort of DTS processing or macro for counting number of "okay" devices"
if "$(interface)" <= "$(total_interfaces)"

config NETWORK_$(interface)_IP_ADDRESS
    string "IP address of network interface $(interface)"

endif

# Kconfig file in test:
interface = 0
source "subsys/Kconfig.network_interfaces"
interface = 1
source "subsys/Kconfig.network_interfaces"
interface = 2
source "subsys/Kconfig.network_interfaces"
interface = 3
source "subsys/Kconfig.network_interfaces"
interface = 4
source "subsys/Kconfig.network_interfaces"

Where total_interfaces would need to be set in dts directly or using macros to count the number of devices of a certain type

[carlescufi]

Arch WG:

• Devicetree vs this new mechanism:
  • @henrikbrixandersen mentions that if there's a tie to a struct device when the property is in Devicetree
  • @fgrandel explains how Devicetree is currently being used in many instances to configure "software"
• @nashif states that we should be able to serialize this in a format that can be used at runtime, independent of the image (i.e. re-provisioning without recompiling)
• @nordicjm states that multi-instance configuration could be done with Kconfig, no need for an additional mechanism to configure this
• @jfischer-no suggests using CBOR as a serialization language
• @henrikbrixandersen states that we may be mixing up different concepts with this: provisioning (serialized data that can be used across multiple instances of a single image) and build-time configuration that is different in every build. @henrikbrixandersen uses the setting subsystem downstream in order to handle this
• @fgrandel reminds us of the provisioning use case for Bluetooth Mesh, which is a very common one. Differentiating between provisioning and multi-instance configuration in terms of language and infrastructure may complicate things, so we may want to have the ability to generate either serialized data or just macros that can be used by the code directly.
• @nashif talks about the developer experience: language, syntax, tooling can all be shared.
• @jfischer-no mentions that anything that resembles a hardware device belongs in Devicetree, and does not require a new mechanism

[de-nordic]

Note on images build for MCUboot:
when planning provisioning mechanism, note that it may be hard to generate some hex and add something directly to image (as part of binary), as outcome of building image for MCUboot may have it already signed and encrypted. So in case when some blob is inserted as binary section, where you could directly address constants, that blob would require signing with the image.

Note on using external, out of image, blob:
such blob will require careful check, the same level as any user input as it may be used as attack vector on firmware. And what to do when these are tampered with? What to revert to? Note that something that may look like tampering may be just a problem that should not brick device, but rather, with annoying effect for user, default to factory setting - the factory settings that provisioning provides.

I have also a question: how to avoid using RAM for things that always be constant for a device? For example you may not change embedded MAC or serial number of device, yet when accessing these, from what has been written during provisioning, the device will have to load them to RAM, at least temporarily. This also brings cost of enabling Flash API and other subsystems required, for device to get the factory provisioned info, even if every unique instance of a device only reads the info and never attempts to store updates, or chooses other subsystem of storage (like FS).

Thanks @carlescufi for subsuming our discussion. A few corrections to avoid misunderstandings:

• @fgrandel explains how Devicetree is currently being used in many instances to configure "software"

Not exactly. I don't use the software/hardware distinction myself. I was rather saying that almost all properties that we introduce change at the same frequency as driver software and not at the same frequency as vendor hardware. From a well-established encapsulation perspective this indicates a requirement to keep driver-specific properties apart from vendor-specific properties and de-facto standardized stable Linux properties (as different files, not as different formats).

I was citing others who had laid out alternative heuristics to define the sw/hw line:

• "Everything that ties to a struct device is 'hardware'" as mentioned by @henrikbrixandersen and others. This heuristic breaks down in both directions: 1. By normalization rules, this definition would include almost all of the network interface configuration which is 1-to-1 to drivers. 2. We have a considerable number of DT properties that do not tie to struct device instances.
• "DT should contain everything that changes at the frequency of hardware" as @decsny (and myself) have been informed by several community members (but does not find himself being practicable nor do I): This heuristic also breaks down in practice. As pointed out above by that definition almost all many Zephyr-specific properties in DT would be "software".

I pointed out that as long as we do not have an intersubjectively observable heuristic that defines the line between sw and hw, each maintainer will define their own personal criteria as we observe in practice. I disagreed with @nashif that we have only few properties that are contended. I'd rather say that almost all newly introduced properties are contended as @decsny has confirmed based on practical experience over the last few months.

• @jfischer-no suggests using CBOR as a serialization language

I mentioned that this is part of the solution space. Others have voted for protobuf or Thrift.

• @henrikbrixandersen states that we may be mixing up different concepts with this: provisioning (serialized data that can be used across multiple instances of a single image) and build-time configuration that is different in every build.

I responded, that we cannot cleanly define per property whether they are "provisioning only" or "build only". On a per-app basis this line will change considerably in practice.

Maybe I misunderstood what was proposed, but if it means that we'd introduce another "a priory" (ontological) distinction then I'd strongly oppose such a strategy. Here a few examples others mentioned in earlier discussions:

• build-time "hw" property (e.g. gain of ADC channels)
• provisioning-time "hw" property (e.g. CDC ECM MAC address)
• build-time "sw" property (e.g. buffer sizes)
• provisioning-time "sw" property (e.g. IPv6 address)

I could immediately think of products (apps) or use cases (tests, debugging) where those properties would not fit those categories.

I therefore argued that any attempt to bind properties to a specific configuration source or target is conceptually flawed and will not work in practice. The fix is: Define a common abstract data model for everything closely tied by encapsulation rules to software programming models (which includes drivers) based on well-established normalization rules (i.e. shared entities and properties) independently from any specific representation or usage context. This can be done w/o any ontological assumptions with mathematical precision and has therefore been well-established best practice in application configuration for decades. This model then needs to be serializable from/to multiple files on a per use-case and encapsulation basis and/or overlayed at any level.

The settings subsystem is part of the solution space but was introduced to the discussion. That's why I voted against having to depend on the settings subsystem for all configuration as it severly limits extensibility of the config subsystem and introduces unnecessary complexity and resource usage to certain use cases like tests or simple custom applications that otherwise won't need it. I vote for technology that allows us to flexibly merge/split serialization from/to any sources/targets depending on specific application requirements. Several technologies with excellent UX exist for all proposed serialization formats to implement such a merge/split from/to a common intermediate representation.

I mentioned that as a bonus, this will make it easier to switch between alternative source and target representations at dev-time (e.g. for easy reproduction of bugs and debugging with minimal dependencies).

• @fgrandel reminds us of the provisioning use case for Bluetooth Mesh

Not only B. Mesh, my comment was including BLE proper as both seem to support the settings subsystem as config target according to @jhedberg. This becomes even more interesting when adding secure storage or large-scale provisioning systems as possible config sources. I also mentioned several other immedately relevant use cases introduced to the discussion by others and that are not yet part of the use case list.

• @nashif talks about the developer experience: language, syntax, tooling can all be shared.

I added to this that IMO tooling will be 90% of the maintenance effort and will be mostly shared between all use cases discussed so far (including init and net config). Assuming a common intermediate representation, sourcing from different formats and serializing into different targets can be added flexibly at very low cost. A common intermediate representation will help to debug the configuration space (similarly to .config and merged DTS) no matter whether done in YAML, JSON or DT.

I therefore vote for extracting and re-using existing DT tooling (not format) and re-use it for whatever configuration system and hopefully common intermediate format we decide upon. Any addition to the DT and/or config world will then equally profit both sides, ie. we can expect that DT tooling will benefit from improvements in the config area, too (e.g. UX improvements, validations, bindings, documentation, etc.). This also has obvious advantages from the user perspective as a lot of target-side and tooling knowledge could be re-used, independently of context.

UPDATE: To clarify as I was probably misunderstood here - I didn't argue for any specific tooling solution here although I admit that I can be read that way. I was just arguing to share tooling among different configuration requirements which is in the problem space, not in the implementation space as I'm not saying anything specific about how such shared tooling should look in the end. OTOH let's be realistic: Any shared tooling will need to build on top of what we have but I admit that such a statement is in the solution space and not required right now. So sorry if I was misread in that sense.

As long as those use cases and basic requirements (including those mentioned in my first comment) are not explicitly made part of the RFC I'll find it very hard to agree to any configuration system. They are all immediately relevant to the initial use cases mentioned by the OP and are therefore part of a "minimal set" as required by this issue. And please note that none of those requirements favors DT over YAML (or whatever other solution space technology) in any way.

UPDATE: Edited to ensure feedback given further down has been included in the argument.

[nashif]

I disagreed with @nashif that we have only few properties that are contended. I'd rather say that almost all newly introduced properties are contended as @decsny has confirmed based on practical experience over the last few months.

you need to provide some context and details here, I have no idea what @decsny said and where he said it and in what context.

[nashif]

I therefore vote for extracting and re-using existing DT tooling (not format) and re-use it for whatever configuration system and hopefully common intermediate format we decide upon.

why would you use a tooling for a format if you are not using the format itself? Also, we established that runtime is important, most of the DT tooling is build time and based on macros.

@nordicjm @de-nordic I think your comments are really relevant but they both rather belong to rather specific solution spaces unless we extract abstract requirements from those examples. I find it hard myself to cleanly distinguish between problem and solution w/o loosing sight of feasibility. But it would be nice if we tried to keep very specific implementation problems always linked to more abstract requirements as I understood @carlescufi 's approach.

@nordicjm In more abstract terms you're making the case - if I understand correctly - that almost any source format will be able to represent an abstract data model (including Kconfig) and that we should not introduce additional formats unless required for a good reason (i.e. simplicity and UX as a requirement).

@de-nordic If I understand you correctly you're requiring at a more abstract level that configuration needs not only to integrate with the settings subsystem alone but also with other subsystems (e.g. MCUBoot signing in your case). From a requirements perspective this confirms that we cannot tie the configuration subsystem target to a single subsystem alone. It must be sufficiently open to integrate with whatever target and process we already have or invent in the future.

@nashif

you need to provide some context and details here, I have no idea what @decsny said and where he said it and in what context.

Sure: #76902 (comment)

[nordicjm]

@nordicjm In more abstract terms you're making the case - if I understand correctly - that almost any source format will be able to represent an abstract data model (including Kconfig) and that we should not introduce additional formats unless required for a good reason (i.e. simplicity and UX as a requirement).

No, I'm stating that without runtime configurability (i.e. provisioning and able to change it) the entire network configuration PR is moot because no tooling is required, static configuration can be wholly done in Kconfig, thus the prime feature that this must support is provisioning