Update docs

cloudinit/cmd/main.py

@@ -949,14 +949,17 @@ def main(sysv_args=None):
         "--debug",
         "-d",
         action="store_true",
-        help="Show additional pre-action logging (default: %(default)s).",
+        help=(
+            "DEPRECATED: Show additional pre-action "
+            "logging (default: %(default)s)."
+        ),
         default=False,
     )
     parser.add_argument(
         "--force",
         action="store_true",
         help=(
-            "Force running even if no datasource is"
+            "DEPRECATED: Force running even if no datasource is"
             " found (use at your own risk)."
         ),
         dest="force",
@@ -979,7 +982,10 @@ def main(sysv_args=None):
 
     # Each action and its sub-options (if any)
     parser_init = subparsers.add_parser(
-        "init", help="Initialize cloud-init and perform initial modules."
+        "init",
+        help=(
+            "DEPRECATED: Initialize cloud-init and perform initial modules."
+        ),
     )
     parser_init.add_argument(
         "--local",
@@ -1002,7 +1008,8 @@ def main(sysv_args=None):
 
     # These settings are used for the 'config' and 'final' stages
     parser_mod = subparsers.add_parser(
-        "modules", help="Activate modules using a given configuration key."
+        "modules",
+        help=("DEPRECATED: Activate modules using a given configuration key."),
     )
     extra_help = lifecycle.deprecate(
         deprecated="`init`",
@@ -1033,7 +1040,11 @@ def main(sysv_args=None):
 
     # This subcommand allows you to run a single module
     parser_single = subparsers.add_parser(
-        "single", help="Run a single module."
+        "single",
+        help=(
+            "Manually run a single module. Useful for "
+            "testing during development."
+        ),
     )
     parser_single.add_argument(
         "--name",

doc/man/cloud-init.1

@@ -4,35 +4,27 @@
 cloud-init \- Cloud instance initialization
 
 .SH SYNOPSIS
-.BR "cloud-init" " [-h] [-d] [-f FILES] [--force] [-v] [SUBCOMMAND]"
+.BR "cloud-init" " [-h] [-d] [--force] [-v] [SUBCOMMAND]"
 
 .SH DESCRIPTION
 Cloud-init provides a mechanism for cloud instance initialization.
 This is done by identifying the cloud platform that is in use, reading
 provided cloud metadata and optional vendor and user
 data, and then initializing the instance as requested.
 
-Generally, this command is not normally meant to be run directly by
-the user. However, some subcommands may useful for development or
-debug of deployments.
-
 .SH OPTIONS
 .TP
 .B "-h, --help"
 Show help message and exit.
 
-.TP
-.B "-d, --debug"
-Show additional pre-action logging (default: False).
-
-.TP
-.B "--force"
-Force running even if no datasource is found (use at your own risk).
-
 .TP
 .B "-v, --version"
 Show program's version number and exit.
 
+.TP
+.B "--all-stages"
+INTERNAL: Run cloud-init's stages under a single process using a syncronization protocol. This is not intended for CLI usage.
+
 .SH SUBCOMMANDS
 Please see the help output for each subcommand for additional details,
 flags, and subcommands.
@@ -57,14 +49,6 @@ Run development tools. See help output for subcommand details.
 .B "features"
 List defined features.
 
-.TP
-.B "init"
-Initialize cloud-init and execute initial modules.
-
-.TP
-.B "modules"
-Activate modules using a given configuration key.
-
 .TP
 .B "query"
 Query standardized instance metadata from the command line.
@@ -75,12 +59,30 @@ Validate cloud-config files using jsonschema.
 
 .TP
 .B "single"
-Run a single module.
+Manually run a single module. Useful for testing during development.
 
 .TP
 .B "status"
 Report cloud-init status or wait on completion.
 
+.SH DEPRECATED
+
+.TP
+.B "-d, --debug"
+Show additional pre-action logging (default: False).
+
+.TP
+.B "--force"
+Force running even if no datasource is found (use at your own risk).
+
+.TP
+.B "init"
+Initialize cloud-init and execute initial modules.
+
+.TP
+.B "modules"
+Activate modules using a given configuration key.
+
 .SH EXIT STATUS
 
 .IP
@@ -95,4 +97,4 @@ Report cloud-init status or wait on completion.
 Copyright (C) 2020 Canonical Ltd. License GPL-3 or Apache-2.0
 
 .SH SEE ALSO
-Full documentation at: <https://cloudinit.readthedocs.io>
+Full documentation at: <https://docs.cloud-init.io>

doc/rtd/explanation/analyze.rst

@@ -3,15 +3,18 @@
 Performance
 ***********
 
-The :command:`analyze` subcommand was added to ``cloud-init`` to help analyze
-``cloud-init`` boot time performance. It is loosely based on
-``systemd-analyze``, where there are four subcommands:
+The :command:`analyze` subcommand helps to analyze ``cloud-init`` boot time
+performance. It is loosely based on ``systemd-analyze``, where there are four
+subcommands:
 
 - :command:`blame`
 - :command:`show`
 - :command:`dump`
 - :command:`boot`
 
+The analyze subcommand works by parsing the cloud-init log file for timestamps
+associated with specific events.
+
 Usage
 =====
 

doc/rtd/explanation/boot.rst

@@ -48,8 +48,7 @@ Detect
 A platform identification tool called ``ds-identify`` runs in the first stage.
 This tool detects which platform the instance is running on. This tool is
 integrated into the init system to disable cloud-init when no platform is
-found, and enable cloud-init when a valid platform is detected. This stage
-might not be present for every installation of cloud-init.
+found, and enable cloud-init when a valid platform is detected.
 
 .. _boot-Local:
 
@@ -88,10 +87,9 @@ is rendered. This includes clearing of all previous (stale) configuration
 including persistent device naming with old MAC addresses.
 
 This stage must block network bring-up or any stale configuration that might
-have already been applied. Otherwise, that could have negative effects such
-as DHCP hooks or broadcast of an old hostname. It would also put the system
-in an odd state to recover from, as it may then have to restart network
-devices.
+have already been applied. Otherwise, that could have negative effects such as
+broadcast of an old hostname. It would also put the system in an odd state to
+recover from, as it may then have to restart network devices.
 
 ``Cloud-init`` then exits and expects for the continued boot of the operating
 system to bring network configuration up as configured.
@@ -189,95 +187,7 @@ finished, the :command:`cloud-init status --wait` subcommand can help block
 external scripts until ``cloud-init`` is done without having to write your own
 ``systemd`` units dependency chains. See :ref:`cli_status` for more info.
 
-.. _boot-First_boot_determination:
-
-First boot determination
-========================
-
-``Cloud-init`` has to determine whether or not the current boot is the first
-boot of a new instance, so that it applies the appropriate configuration. On
-an instance's first boot, it should run all "per-instance" configuration,
-whereas on a subsequent boot it should run only "per-boot" configuration. This
-section describes how ``cloud-init`` performs this determination, as well as
-why it is necessary.
-
-When it runs, ``cloud-init`` stores a cache of its internal state for use
-across stages and boots.
-
-If this cache is present, then ``cloud-init`` has run on this system
-before [#not-present]_. There are two cases where this could occur. Most
-commonly, the instance has been rebooted, and this is a second/subsequent
-boot. Alternatively, the filesystem has been attached to a *new* instance,
-and this is the instance's first boot. The most obvious case where this
-happens is when an instance is launched from an image captured from a
-launched instance.
-
-By default, ``cloud-init`` attempts to determine which case it is running
-in by checking the instance ID in the cache against the instance ID it
-determines at runtime. If they do not match, then this is an instance's
-first boot; otherwise, it's a subsequent boot. Internally, ``cloud-init``
-refers to this behaviour as ``check``.
-
-This behaviour is required for images captured from launched instances to
-behave correctly, and so is the default that generic cloud images ship with.
-However, there are cases where it can cause problems [#problems]_. For these
-cases, ``cloud-init`` has support for modifying its behaviour to trust the
-instance ID that is present in the system unconditionally. This means that
-``cloud-init`` will never detect a new instance when the cache is present,
-and it follows that the only way to cause ``cloud-init`` to detect a new
-instance (and therefore its first boot) is to manually remove
-``cloud-init``'s cache. Internally, this behaviour is referred to as
-``trust``.
-
-To configure which of these behaviours to use, ``cloud-init`` exposes the
-``manual_cache_clean`` configuration option.  When ``false`` (the default),
-``cloud-init`` will ``check`` and clean the cache if the instance IDs do
-not match (this is the default, as discussed above). When ``true``,
-``cloud-init`` will ``trust`` the existing cache (and therefore not clean it).
-
-Manual cache cleaning
-=====================
-
-``Cloud-init`` ships a command for manually cleaning the cache:
-:command:`cloud-init clean`. See :ref:`cli_clean`'s documentation for further
-details.
-
-Reverting ``manual_cache_clean`` setting
-----------------------------------------
-
-Currently there is no support for switching an instance that is launched with
-``manual_cache_clean: true`` from ``trust`` behaviour to ``check`` behaviour,
-other than manually cleaning the cache.
-
-.. warning:: If you want to capture an instance that is currently in ``trust``
-   mode as an image for launching other instances, you **must** manually clean
-   the cache. If you do not do so, then instances launched from the captured
-   image will all detect their first boot as a subsequent boot of the captured
-   instance, and will not apply any per-instance configuration.
-
-   This is a functional issue, but also a potential security one:
-   ``cloud-init`` is responsible for rotating SSH host keys on first boot,
-   and this will not happen on these instances.
-
-.. [#not-present] It follows that if this cache is not present,
-   ``cloud-init`` has not run on this system before, so this is
-   unambiguously this instance's first boot.
-
-.. [#problems] A couple of ways in which this strict reliance on the presence
-   of a datasource has been observed to cause problems:
-
-    - If a cloud's metadata service is flaky and ``cloud-init`` cannot
-      obtain the instance ID locally on that platform, ``cloud-init``'s
-      instance ID determination will sometimes fail to determine the current
-      instance ID, which makes it impossible to determine if this is an
-      instance's first or subsequent boot (`#1885527`_).
-    - If ``cloud-init`` is used to provision a physical appliance or device
-      and an attacker can present a datasource to the device with a different
-      instance ID, then ``cloud-init``'s default behaviour will detect this as
-      an instance's first boot and reset the device using the attacker's
-      configuration (this has been observed with the
-      :ref:`NoCloud datasource<datasource_nocloud>` in `#1879530`_).
+See the :ref:`first boot documentation <First_boot_determination>` to learn how
+cloud-init decides that a boot is the "first boot".
 
 .. _generator: https://www.freedesktop.org/software/systemd/man/systemd.generator.html
-.. _#1885527: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1885527
-.. _#1879530: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1879530

doc/rtd/explanation/events.rst

@@ -66,9 +66,10 @@ Hotplug
 =======
 
 When the ``hotplug`` event is supported by the datasource and configured in
-user data, ``cloud-init`` will respond to the addition or removal of network
-interfaces to the system. In addition to fetching and updating the system
-metadata, ``cloud-init`` will also bring up/down the newly added interface.
+:ref:`user data<mod_cc_install_hotplug>`, ``cloud-init`` will respond to the
+addition or removal of network interfaces to the system. In addition to
+fetching and updating the system metadata, ``cloud-init`` will also bring
+up/down the newly added interface.
 
 .. warning::
    Due to its use of ``systemd`` sockets, ``hotplug`` functionality is

doc/rtd/explanation/first_boot.rst

@@ -0,0 +1,91 @@
+.. _First_boot_determination:
+
+First boot determination
+========================
+
+``Cloud-init`` has to determine whether or not the current boot is the first
+boot of a new instance, so that it applies the appropriate configuration. On
+an instance's first boot, it should run all "per-instance" configuration,
+whereas on a subsequent boot it should run only "per-boot" configuration. This
+section describes how ``cloud-init`` performs this determination, as well as
+why it is necessary.
+
+When it runs, ``cloud-init`` stores a cache of its internal state for use
+across stages and boots.
+
+If this cache is present, then ``cloud-init`` has run on this system
+before [#not-present]_. There are two cases where this could occur. Most
+commonly, the instance has been rebooted, and this is a second/subsequent
+boot. Alternatively, the filesystem has been attached to a *new* instance,
+and this is the instance's first boot. The most obvious case where this
+happens is when an instance is launched from an image captured from a
+launched instance.
+
+By default, ``cloud-init`` attempts to determine which case it is running
+in by checking the instance ID in the cache against the instance ID it
+determines at runtime. If they do not match, then this is an instance's
+first boot; otherwise, it's a subsequent boot. Internally, ``cloud-init``
+refers to this behaviour as ``check``.
+
+This behaviour is required for images captured from launched instances to
+behave correctly, and so is the default that generic cloud images ship with.
+However, there are cases where it can cause problems [#problems]_. For these
+cases, ``cloud-init`` has support for modifying its behaviour to trust the
+instance ID that is present in the system unconditionally. This means that
+``cloud-init`` will never detect a new instance when the cache is present,
+and it follows that the only way to cause ``cloud-init`` to detect a new
+instance (and therefore its first boot) is to manually remove
+``cloud-init``'s cache. Internally, this behaviour is referred to as
+``trust``.
+
+To configure which of these behaviours to use, ``cloud-init`` exposes the
+``manual_cache_clean`` configuration option.  When ``false`` (the default),
+``cloud-init`` will ``check`` and clean the cache if the instance IDs do
+not match (this is the default, as discussed above). When ``true``,
+``cloud-init`` will ``trust`` the existing cache (and therefore not clean it).
+
+Manual cache cleaning
+=====================
+
+``Cloud-init`` ships a command for manually cleaning the cache:
+:command:`cloud-init clean`. See :ref:`cli_clean`'s documentation for further
+details.
+
+Reverting ``manual_cache_clean`` setting
+----------------------------------------
+
+Currently there is no support for switching an instance that is launched with
+``manual_cache_clean: true`` from ``trust`` behaviour to ``check`` behaviour,
+other than manually cleaning the cache.
+
+.. warning:: If you want to capture an instance that is currently in ``trust``
+   mode as an image for launching other instances, you **must** manually clean
+   the cache. If you do not do so, then instances launched from the captured
+   image will all detect their first boot as a subsequent boot of the captured
+   instance, and will not apply any per-instance configuration.
+
+   This is a functional issue, but also a potential security one:
+   ``cloud-init`` is responsible for rotating SSH host keys on first boot,
+   and this will not happen on these instances.
+
+.. [#not-present] It follows that if this cache is not present,
+   ``cloud-init`` has not run on this system before, so this is
+   unambiguously this instance's first boot.
+
+.. [#problems] A couple of ways in which this strict reliance on the presence
+   of a datasource has been observed to cause problems:
+
+    - If a cloud's metadata service is flaky and ``cloud-init`` cannot
+      obtain the instance ID locally on that platform, ``cloud-init``'s
+      instance ID determination will sometimes fail to determine the current
+      instance ID, which makes it impossible to determine if this is an
+      instance's first or subsequent boot (`#1885527`_).
+    - If ``cloud-init`` is used to provision a physical appliance or device
+      and an attacker can present a datasource to the device with a different
+      instance ID, then ``cloud-init``'s default behaviour will detect this as
+      an instance's first boot and reset the device using the attacker's
+      configuration (this has been observed with the
+      :ref:`NoCloud datasource<datasource_nocloud>` in `#1879530`_).
+
+.. _#1885527: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1885527
+.. _#1879530: https://bugs.launchpad.net/ubuntu/+source/cloud-init/+bug/1879530