[rfc]: Add support for allowing user-defined network configuration at runtime

[holmanb]

How the network is configured today

In many cases, manually configuring an instance's network configuration on boot is unnecessary. This is handled automatically by cloud-init in the following way:

Cloud-init automatically configures the network when a cloud image boots. To accomplish this, it provides a reasonable default network configuration to the system's network manager (systemd-networkd, NetworkManager, netplan, eni, sysconfig, etc).

In some cases, the cloud may request that the network be configured in a specific way at runtime. This allows cloud providers to support a more dynamic network environment. Cloud-init accomplishes this by setting up a temporary network to read a configuration[1] from the cloud's IMDS server. This all happens before the networking manager is started by the init system, so no service restarts or reconfiguration is necessary.

The cloud-provided configuration may be overridden by the user in the base image by modifying /etc/cloud/cloud.cfg.d/*, or in the kernel command line. This doesn't, however, allow the user to manually set the network configuration at runtime.

More details on network configuration can be found here.

Allowing user-defined network configuration at runtime

Despite cloud-init configuring networking in a way that accomplishes the users' needs in many cases, some users might wish to customize their instance with a configuration at runtime. This might be provided in the same way that users provide cloud-config user-data (perhaps as a new sub-key?).

Such a feature would be quite powerful. It would also be difficult to debug (though some clouds provide a way to "serial console" into the instance for debugging). Since some clouds expect to be able to provide dynamic configuration via cloud metadata, a user that overrides this configuration with a custom one might unexpectedly prevent the cloud from delivering the configuration that is needed to deal with a dynamic network environment. This means that if cloud-init grows the ability to allow users to provide network configuration, this would require that either 1) users are expected to know what they are doing or 2) we provide some safety net, such as the ability for the cloud-provided configuration to (optionally) merge with the user-provided configuration. At a minimum, we would have to document which clouds provide networking configuration.

Example use cases (add more in comments below please)

• [enhancement]: Add ability to pass specific ipv4/ipv6 options to network renderers #4621 (comment)

[1] the configuration formats[1][2] are converted (rendered)

[jsf9k]

FWIW, I don't see anything in the AWS EC2 IMDS about DHCP.

[holmanb]

Continued from the original comment in a separate issue: #4621 (comment)

It sounds like I'll have to use runcmd to edit the /etc/netplan/50-cloud-init.yaml file that cloud-init produces, then run netplan generate. Do you see any other workaround?

You probably want netplan apply actually. The command netplan generate will generate the configuration that is used by networkd/networkmanager, but won't actually configure the network. For now, this is probably the best workaround.

[holmanb]

FWIW, I don't see anything in the AWS EC2 IMDS about DHCP.

Agreed, but in cloud-init the capability already exists to set this (via netplan / networkv2) - it's just not possible for users to set this at runtime.

[aciba90]

This might be a way to fix #5249.