doc: Describe all stages in a single process (#5595)

doc/man/cloud-init.1

@@ -4,35 +4,27 @@
 cloud-init \- Cloud instance initialization
 
 .SH SYNOPSIS
-.BR "cloud-init" " [-h] [-d] [-f FILES] [--force] [-v] [SUBCOMMAND]"
+.BR "cloud-init" " [-h] [-d] [--force] [-v] [SUBCOMMAND]"
 
 .SH DESCRIPTION
 Cloud-init provides a mechanism for cloud instance initialization.
 This is done by identifying the cloud platform that is in use, reading
 provided cloud metadata and optional vendor and user
 data, and then initializing the instance as requested.
 
-Generally, this command is not normally meant to be run directly by
-the user. However, some subcommands may useful for development or
-debug of deployments.
-
 .SH OPTIONS
 .TP
 .B "-h, --help"
 Show help message and exit.
 
-.TP
-.B "-d, --debug"
-Show additional pre-action logging (default: False).
-
-.TP
-.B "--force"
-Force running even if no datasource is found (use at your own risk).
-
 .TP
 .B "-v, --version"
 Show program's version number and exit.
 
+.TP
+.B "--all-stages"
+INTERNAL: Run cloud-init's stages under a single process using a syncronization protocol. This is not intended for CLI usage.
+
 .SH SUBCOMMANDS
 Please see the help output for each subcommand for additional details,
 flags, and subcommands.
@@ -57,14 +49,6 @@ Run development tools. See help output for subcommand details.
 .B "features"
 List defined features.
 
-.TP
-.B "init"
-Initialize cloud-init and execute initial modules.
-
-.TP
-.B "modules"
-Activate modules using a given configuration key.
-
 .TP
 .B "query"
 Query standardized instance metadata from the command line.
@@ -75,12 +59,30 @@ Validate cloud-config files using jsonschema.
 
 .TP
 .B "single"
-Run a single module.
+Manually run a single module. Useful for testing during development.
 
 .TP
 .B "status"
 Report cloud-init status or wait on completion.
 
+.SH DEPRECATED
+
+.TP
+.B "-d, --debug"
+Show additional pre-action logging (default: False).
+
+.TP
+.B "--force"
+Force running even if no datasource is found (use at your own risk).
+
+.TP
+.B "init"
+Initialize cloud-init and execute initial modules.
+
+.TP
+.B "modules"
+Activate modules using a given configuration key.
+
 .SH EXIT STATUS
 
 .IP
@@ -95,4 +97,4 @@ Report cloud-init status or wait on completion.
 Copyright (C) 2020 Canonical Ltd. License GPL-3 or Apache-2.0
 
 .SH SEE ALSO
-Full documentation at: <https://cloudinit.readthedocs.io>
+Full documentation at: <https://docs.cloud-init.io>

doc/rtd/explanation/boot.rst

@@ -48,8 +48,7 @@ Detect
 A platform identification tool called ``ds-identify`` runs in the first stage.
 This tool detects which platform the instance is running on. This tool is
 integrated into the init system to disable cloud-init when no platform is
-found, and enable cloud-init when a valid platform is detected. This stage
-might not be present for every installation of cloud-init.
+found, and enable cloud-init when a valid platform is detected.
 
 .. _boot-Local:
 
@@ -88,10 +87,9 @@ is rendered. This includes clearing of all previous (stale) configuration
 including persistent device naming with old MAC addresses.
 
 This stage must block network bring-up or any stale configuration that might
-have already been applied. Otherwise, that could have negative effects such
-as DHCP hooks or broadcast of an old hostname. It would also put the system
-in an odd state to recover from, as it may then have to restart network
-devices.
+have already been applied. Otherwise, that could have negative effects such as
+broadcast of an old hostname. It would also put the system in an odd state to
+recover from, as it may then have to restart network devices.
 
 ``Cloud-init`` then exits and expects for the continued boot of the operating
 system to bring network configuration up as configured.

doc/rtd/howto/rerun_cloud_init.rst

@@ -64,33 +64,31 @@ a result.
 
 .. _partially_rerun_cloud_init:
 
-How to partially re-run cloud-init
-==================================
-
-If the behavior you are testing runs on every boot, there are a couple
-of ways to test this behavior.
-
 Manually run cloud-init stages
 ------------------------------
 
-Note that during normal boot of cloud-init, the init system runs these
-stages at specific points during boot. This means that running the code
-manually after booting the system may cause the code to interact with
-the system in a different way than it does while it boots.
+During normal boot of cloud-init, the init system runs the following command
+command:
 
 .. code-block:: shell-session
 
-   cloud-init init --local
-   cloud-init init
-   cloud-init modules --mode=config
-   cloud-init modules --mode=final
+   cloud-init --all-stages
+
+Keep in mind that running this manually may not behave the same as cloud-init
+behaves when it is started by the init system. The first reason for this is
+that cloud-init's stages are intended to run before and after specific events
+in the boot order, so there are no guarantees that it will do the right thing
+when running out of order. The second reason is that cloud-init will skip its
+normal synchronization protocol when it detects that stdin is a tty for purpose
+of debugging and development.
+
+This command cannot be expected to be stable when executed outside of the init
+system due to its ordering requirements.
 
 Reboot the instance
 -------------------
 
-Rebooting the instance will take a little bit longer, however it will
-make cloud-init stages run at the correct times during boot, so it will
-behave more correctly.
+Rebooting the instance will re-run any parts of cloud-init that run per-boot.
 
 .. code-block:: shell-session
 

doc/rtd/reference/breaking_changes.rst

@@ -11,6 +11,45 @@ releases.
     many operating system vendors patch out breaking changes in
     cloud-init to ensure consistent behavior on their platform.
 
+24.3
+====
+
+Single Process Optimization
+---------------------------
+
+As a performance optimization, cloud-init no longer runs as four seperate
+Python processes. Instead, it launches a single process and then
+communicates with the init system over a Unix socket to allow the init system
+to tell it when it should start each stage and to tell the init system when
+each stage has completed. Init system ordering is preserved.
+
+This should have no noticable affect for end users, besides a faster boot time.
+This is a breaking change for two reasons:
+
+1. a precaution to avoid unintentionally breaking users on stable distributions
+2. this change included renaming a systemd service:
+   ``cloud-init.service`` -> ``cloud-init-network.service``
+
+The now-deprecated command line arguments used to invoke each stage will still
+be supported for a period of time to allow for adoption and stabilization. Any
+systemd distribution that wants to revert this behavior may want to
+`patch this change`_.
+
+Support has not yet been added for non-systemd distributions, however it is
+possible to add support.
+
+Note that this change adds dependency on the openbsd netcat implementation,
+which is already on Ubuntu as part of ``ubuntu-minimal``.
+
+Addition of NoCloud network-config
+----------------------------------
+
+The NoCloud datasource now has support for providing network configuration
+using network-config. Any installation that doesn't provide this configuration
+file will experience a retry/timeout in boot. Adding an empty
+``network-config`` file should provide backwards compatibility with previous
+behavior.
+
 24.1
 ====
 
@@ -96,3 +135,6 @@ behavior as a result of this change.
 
 Workarounds include updating the kernel command line and optionally configuring
 a ``datasource_list`` in ``/etc/cloud/cloud.cfg.d/*.cfg``.
+
+
+.. _patch this change: https://github.com/canonical/cloud-init/blob/ubuntu/noble/debian/patches/no-single-process.patch

doc/rtd/reference/cli.rst

@@ -15,20 +15,20 @@ Example output:
 
 .. code-block::
 
-   usage: cloud-init [-h] [--version] [--debug] [--force]
-                                                               {init,modules,single,query,features,analyze,devel,collect-logs,clean,status,schema} ...
+   usage: cloud-init [-h] [--version] [--debug] [--force] [--all-stages] {init,modules,single,query,features,analyze,devel,collect-logs,clean,status,schema} ...
 
     options:
       -h, --help            show this help message and exit
       --version, -v         Show program's version number and exit.
       --debug, -d           Show additional pre-action logging (default: False).
       --force               Force running even if no datasource is found (use at your own risk).
+      --all-stages          Run cloud-init's stages under a single process using a syncronization protocol. This is not intended for CLI usage.
 
     Subcommands:
       {init,modules,single,query,features,analyze,devel,collect-logs,clean,status,schema}
-        init                Initialize cloud-init and perform initial modules.
-        modules             Activate modules using a given configuration key.
-        single              Run a single module.
+        init                DEPRECATED: Initialize cloud-init and perform initial modules.
+        modules             DEPRECATED: Activate modules using a given configuration key.
+        single              Manually run a single module. Useful for testing during development.
         query               Query standardized instance metadata from the command line.
         features            List defined features.
         analyze             Devel tool: Analyze cloud-init logs and data.
@@ -185,8 +185,8 @@ Example output:
 
 .. _cli_init:
 
-:command:`init`
-===============
+:command:`init` (deprecated)
+============================
 
 Generally run by OS init systems to execute ``cloud-init``'s stages:
 *init* and *init-local*. See :ref:`boot_stages` for more info.
@@ -200,8 +200,8 @@ generally gated to run only once due to semaphores in
 
 .. _cli_modules:
 
-:command:`modules`
-==================
+:command:`modules` (deprecated)
+===============================
 
 Generally run by OS init systems to execute ``modules:config`` and
 ``modules:final`` boot stages. This executes cloud config :ref:`modules`