diff --git a/installing/installing_bare_metal/installing-bare-metal-network-customizations.adoc b/installing/installing_bare_metal/installing-bare-metal-network-customizations.adoc index 92535ad09466..84e4cef22d8f 100644 --- a/installing/installing_bare_metal/installing-bare-metal-network-customizations.adoc +++ b/installing/installing_bare_metal/installing-bare-metal-network-customizations.adoc @@ -59,12 +59,48 @@ Before you install a cluster on bare metal infrastructure that you provision, you must create {op-system} machines for it to use. Follow either the steps to use an ISO image or network PXE booting to create the machines. -include::modules/installation-user-infra-machines-iso.adoc[leveloffset=+2] +There are several methods of configuring {op-system} during ISO and +PXE installations. These include: + +* Kernel arguments: For a PXE install, you can `APPEND` arguments to the +kernel of the live installer. For an ISO install, you can interrupt the +live install boot process to add kernel arguments. In both cases, you can use +special `coreos-inst*` arguments to direct the installer, as well as +standard boot arguments for turning standard kernel services +on or off. + +* Ignition configs: You need to generate an {product-title} Ignition config +(*.ign) file for the type of node you are installing (worker, control plane, +or bootstrap). You pass the location of the Ignition config to the installed +system so that it takes effect on first boot. You can also embed that Ignition +Config into the ISO before you boot it. In special cases, you can create a +separate, limited Ignition config to pass to the live system. That limited +Ignition config could do a limited +set of tasks, like reporting reporting success to a provisioning system +after completing installation. +This special Ignition config is consumed by the installer and should not +be used to include the standard `worker` and `master` Ignition configs. + +* coreos-installer: You can boot the live ISO installer to a shell prompt, +which allows you to prepare the permanent system in a variety of ways +before first boot. In particular, you can manually run the `coreos-installer` +command from a shell prompt, passing it options to configure +some details of the installed system. + +Whether to use an ISO or PXE install depends on your situation. +A PXE install requires an available DHCP service and more preparation, +but can make the installation process more automated. An ISO install +is a more manual process and can be inconvenient if you are setting +up more than a few machines. -include::modules/installation-user-infra-machines-static-network.adoc[leveloffset=+3] +include::modules/installation-user-infra-machines-iso.adoc[leveloffset=+2] include::modules/installation-user-infra-machines-pxe.adoc[leveloffset=+2] +include::modules/installation-user-infra-machines-advanced.adoc[leveloffset=+2] + +include::modules/installation-user-infra-machines-static-network.adoc[leveloffset=+3] + include::modules/installation-installing-bare-metal.adoc[leveloffset=+1] include::modules/cli-logging-in-kubeadmin.adoc[leveloffset=+1] diff --git a/installing/installing_bare_metal/installing-bare-metal.adoc b/installing/installing_bare_metal/installing-bare-metal.adoc index 6f796c38ceb2..3e21ff4eebe3 100644 --- a/installing/installing_bare_metal/installing-bare-metal.adoc +++ b/installing/installing_bare_metal/installing-bare-metal.adoc @@ -63,12 +63,52 @@ Before you install a cluster on bare metal infrastructure that you provision, you must create {op-system} machines for it to use. Follow either the steps to use an ISO image or network PXE booting to create the machines. -include::modules/installation-user-infra-machines-iso.adoc[leveloffset=+2] +There are several methods of configuring {op-system} during ISO and +PXE installations. These include: + +* Kernel arguments: For a PXE install, you can `APPEND` arguments to the +kernel of the live installer. For an ISO install, you can interrupt the +live install boot process to add kernel arguments. In both cases, you can use +`special coreos-inst*` arguments, to direct the live installer, as well as +standard installation boot arguments, for turning standard kernel services +on or off. + +* Ignition configs: You need to generate an {product-title} Ignition config +(*.ign) file for the type of node you are installing (worker, control plane, +or bootstrap). You pass the location of the Ignition config to the installed +system, so it takes effect on first boot. You can also embed that Ignition +Config into the ISO before you boot it. In special cases, you can create a +separate Ignition config to pass to the live system, to do things like disk +configuration before installing the system. + +* coreos-installer: You can boot the live ISO installer to a shell prompt, +which allows you to prepare the permanent system in a variety of ways +before first boot. In particular, you can run the `coreos-installer` +command to identify various artifacts to include, work with disk partitions, +and set up networking. In some cases, you can configure features on +the live system and copy them to the installed system. + +Whether to use an ISO or PXE install depends on your situation. +A PXE install requires an available DHCP service and more preparation, +but can make the installation process more automated. An ISO install +is a more manual process and can be inconvenient if you are setting +up more than a few machines. + +[NOTE] +==== +As of {product-title} 4.6, the {op-system} ISO and other installation artifacts +provide support for installation on disks with 4k sectors. +==== -include::modules/installation-user-infra-machines-static-network.adoc[leveloffset=+3] + +include::modules/installation-user-infra-machines-iso.adoc[leveloffset=+2] include::modules/installation-user-infra-machines-pxe.adoc[leveloffset=+2] +include::modules/installation-user-infra-machines-advanced.adoc[leveloffset=+2] + +include::modules/installation-user-infra-machines-static-network.adoc[leveloffset=+3] + include::modules/installation-installing-bare-metal.adoc[leveloffset=+1] include::modules/cli-logging-in-kubeadmin.adoc[leveloffset=+1] diff --git a/installing/installing_bare_metal/installing-restricted-networks-bare-metal.adoc b/installing/installing_bare_metal/installing-restricted-networks-bare-metal.adoc index 5e9ba00ac9a4..7bed568cb2f8 100644 --- a/installing/installing_bare_metal/installing-restricted-networks-bare-metal.adoc +++ b/installing/installing_bare_metal/installing-restricted-networks-bare-metal.adoc @@ -76,12 +76,45 @@ Before you install a cluster on bare metal infrastructure that you provision, you must create {op-system} machines for it to use. Follow either the steps to use an ISO image or network PXE booting to create the machines. -include::modules/installation-user-infra-machines-iso.adoc[leveloffset=+2] +There are several methods of configuring {op-system} during ISO and +PXE installations. These include: + +* Kernel arguments: For a PXE install, you can `APPEND` arguments to the +kernel of the live installer. For an ISO install, you can interrupt the +live install boot process to add kernel arguments. In both cases, you can use +`special coreos-inst*` arguments, to direct the live installer, as well as +standard installation boot arguments, for turning standard kernel services +on or off. + +* Ignition configs: You need to generate an {product-title} Ignition config +(*.ign) file for the type of node you are installing (worker, control plane, +or bootstrap). You pass the location of the Ignition config to the installed +system, so it takes effect on first boot. You can also embed that Ignition +Config into the ISO before you boot it. In special cases, you can create a +separate Ignition config to pass to the live system, to do things like disk +configuration before installing the system. + +* coreos-installer: You can boot the live ISO installer to a shell prompt, +which allows you to prepare the permanent system in a variety of ways +before first boot. In particular, you can run the `coreos-installer` +command to identify various artifacts to include, work with disk partitions, +and set up networking. In some cases, you can configure features on +the live system and copy them to the installed system. + +Whether to use an ISO or PXE install depends on your situation. +A PXE install requires an available DHCP service and more preparation, +but can make the installation process more automated. An ISO install +is a more manual process and can be inconvenient if you are setting +up more than a few machines. -include::modules/installation-user-infra-machines-static-network.adoc[leveloffset=+3] +include::modules/installation-user-infra-machines-iso.adoc[leveloffset=+2] include::modules/installation-user-infra-machines-pxe.adoc[leveloffset=+2] +include::modules/installation-user-infra-machines-advanced.adoc[leveloffset=+2] + +include::modules/installation-user-infra-machines-static-network.adoc[leveloffset=+3] + include::modules/installation-installing-bare-metal.adoc[leveloffset=+1] include::modules/cli-logging-in-kubeadmin.adoc[leveloffset=+1] diff --git a/modules/digging-into-machine-config.adoc b/modules/digging-into-machine-config.adoc index 6f32d06e0564..75eed62608c0 100644 --- a/modules/digging-into-machine-config.adoc +++ b/modules/digging-into-machine-config.adoc @@ -3,7 +3,7 @@ // * architecture/architecture_rhcos.adoc [id="digging-into-machine-config_{context}"] -= Changing Ignition Configs after installation += Changing Ignition configs after installation Machine Config Pools manage a cluster of nodes and their corresponding Machine Configs. Machine Configs contain configuration information for a cluster. diff --git a/modules/installation-user-infra-machines-advanced.adoc b/modules/installation-user-infra-machines-advanced.adoc new file mode 100644 index 000000000000..f23a2837be94 --- /dev/null +++ b/modules/installation-user-infra-machines-advanced.adoc @@ -0,0 +1,339 @@ +// Module included in the following assemblies: +// +// * installing/installing_bare_metal/installing-bare-metal.adoc +// * installing/installing_bare_metal/installing-restricted-networks-bare-metal.adoc +// * installing_bare_metal/installing-bare-metal-network-customizations.adoc + +[id="installation-user-infra-machines-advanced_{context}"] += Advanced {op-system-first} installation configuration + +A key reason for doing a bare metal installation of {op-system-first} +nodes for {product-title} is to be able to do configuration that is not +available through default {product-title} installation methods. +This section describes some of the configurations you can do using +techniques that include: + +* Passing kernel arguments to the live installer +* Running the coreos-installer during a live installation +* Embedding Ignition configs in an ISO + +The most useful advanced configuration topics for bare metal {op-system-first} +installations relate to disk partitioning, networking, and using Ignition configs in different ways. + +[id="installation-user-infra-machines-advanced_network_{context}"] +== Networking +Networking for {product-title} nodes uses DHCP by default to gather all +necessary configuration settings. If you want to set up static IP addresses +or configure special settings, such as bonding, you can do that by either: + +* Passing special kernel parameters when you boot the live installer, +* Using MachineConfigs to copy networking files to the installed system, or +* Configuring networking from a live installer shell prompt, then copying +those settings to the installed system so they take effect when the +installed system first boots. + +For a PXE or iPXE installation, see the `Configure advanced networking` table for network-related kernel arguments to APPEND to the kernel or use MachineConfigs to copy networking files to the installed system. + +For an ISO installation, do the following: + +.Procedure + +. Boot the ISO installer. +. From the live system shell prompt, configure networking for the live +system using available RHEL tools, such as `nmcli` or `nmtui`. +. Include the `--copy-network option` with the options when you run the +`coreos-installer` command. For example: ++ +[source,terminal] +---- +$ coreos-installer install --copy-network \ + --ignition-url=http://host/worker.ign /dev/sda +---- + +. Once the installer is done, boot up to the system you installed to disk, which will include the new network settings. + +[id="installation-user-infra-machines-advanced_disk_{context}"] +== Disk partitioning +In most cases, the {product-title} installer should be allowed to configure +your disk partitions. However, here are two cases where you might want to +intervene to override the default partitioning when installing an +{product-title} node: + +* Creating separate partitions: For greenfield installations on an empty +disk, you may want to add separate storage to a partition. This is only +officially supported for making `/var` or a subdirectory of `/var` a separate +partition (not both). If you create more than one partitoin, Kubernetes +will not be able to monitor them both. + +* Retaining existing partitions: For a brownfield installation, where you +are reinstalling on an existing {product-title} node, or a greenfield +installation where you just want to partition a disk manually before +starting the installation, there are both boot arguments and options to +`coreos-installer` that let you retain existing data partitions. + +[id="installation-user-infra-machines-advanced_vardisk_{context}"] +=== Creating a separate /var partition +In general, disk partitioning for {product-title} should be left to the +installer. However, there are cases where you might want to create a +separate partition in a part of the filesystem that you expect to grow. + +{product-title} supports the addition of a single partition to attach +storage to either the `/var` partition or a subdirectory of `/var`. +For example: + +* `/var/lib/containers`: Holds container-related content that can grow +as more images and containers are added to a system. +* `/var/log`: Holds logging data that you might want to keep separate for +auditing purposes later. + +Storing the contents of a `/var` directory separately lets you more easily +grow storage to those areas as needed and possibly reinstall {product-title} +at a later date and keep that data intact. In other words, you would not have +to pull all your containers again or copy off massive log files when you +update your systems. + +Because `/var` must be in place before a fresh installation of +{op-system-first}, this procedure sets up the separate `/var` partition +by creating a MachineConfig that is inserted during the `openshift-install` +preparation phases of an {product-title} installation. + +.Procedure + +. Create a directory to hold the {product-title} installation files: ++ +[source,terminal] +---- +$ mkdir $HOME/clusterconfig +---- + +. Run `openshift-install` to create a set of files in the `manifest` and +`openshift` subdirectories. Answer questions as prompted: ++ +[source,terminal] +---- +$ openshift-install create manifests --dir $HOME/clusterconfig +? SSH Public Key ... +$ ls $HOME/clusterconfig/openshift/ +99_kubeadmin-password-secret.yaml +99_openshift-cluster-api_master-machines-0.yaml +99_openshift-cluster-api_master-machines-1.yaml +99_openshift-cluster-api_master-machines-2.yaml +... +---- + +. Create a MachineConfig and add it to a file in the `openshift` directory. +For example, name the file `98-var-log-partition.yaml`, +changing the device name to the name of the storage device on the `worker` systems +and set storage size as appropriate. It will attach storage to a separate `/var/log` +directory. + ++ +[source,terminal] +---- +apiVersion: machineconfiguration.openshift.io/v1 +kind: MachineConfig +metadata: + labels: + machineconfiguration.openshift.io/role: worker + name: 98-var-log-partition +spec: + config: + ignition: + version: 3.1.0 + storage: + disks: + - device: /dev/nvme0n1 + wipeTable: false + partitions: + - sizeMiB: 47000 + startMiB: 47000 + label: var-log + filesystems: + - path: /var/log + device: /dev/disk/by-partlabel/var-log + format: xfs + systemd: + units: + - name: var-log.mount + enabled: true + contents: | + [Unit] + Before=local-fs.target + [Mount] + Where=/var/log + What=/dev/disk/by-partlabel/var-log + [Install] + WantedBy=local-fs.target +---- + +. Run `openshift-install` again to create Ignition configs from a set of files in the `manifest` and +`openshift` subdirectories: ++ +[source,terminal] +---- +$ openshift-install create ignition-configs --dir $HOME/clusterconfig +$ ls $HOME/clusterconfig/ +auth bootstrap.ign master.ign metadata.json worker.ign +---- + +At this point, you can use the Ignition config files as input to the ISO or PXE bare +metal installation procedures to install {op-system-first} systems. + +[id="installation-user-infra-machines-advanced_retaindisk_{context}"] +=== Retaining existing partitions +For an ISO installation, you can add options to the `coreos-installer` +that causes the installer to maintain one or more existing partitions. +For a PXE installation, you can APPEND coreos.inst options to preserve partitions. + +Saved partitions might be partitions from an existing {product-title} +system that has data partitions that you want to keep or partitions +that you just manually created. Here are a few tips: + +* Make sure you assign at least the recommended amount of disk space to the +OpenShift partitions at the beginning of the disk. + +* Identify the disk partitions you want to keep either by partition +number or label. + +For an ISO installation: + +The following example illustrates running the coreos-installer in a way that preserves +the sixth (6) partition on the disk: + +[source,terminal] +---- +# coreos-installer install --ignition-url http://10.0.2.2:8080/user.ign \ + --save-partindex 6 /dev/sda +---- + +This example preserves partitions 5 and higher: + +[source,terminal] +---- +# coreos-installer install --ignition-url http://10.0.2.2:8080/user.ign + --save-partindex 5- /dev/sda +---- + +This example preserves any partition in which the partition label begins with `data` (`data*`): + +[source,terminal] +---- +# coreos-installer install --ignition-url http://10.0.2.2:8080/user.ign + --save-partlabel ‘data*’ /dev/sda +---- + +In all of those examples, after the install completes, on the first boot of +the new system, Ignition will recreate the partition at the same place it +was before (the same offset), and the filesystem will be intact. + +For a PXE installation: + +This APPEND option preserves the sixth partition: + +[source,terminal] +---- +coreos.inst.save_part=6 +---- + +This APPEND option preserves the all partitions from 5 and higher: + +[source,terminal] +---- +coreos.inst.save_part=5- +---- + +This APPEND option preserves any partition in which the partition label begins with `data` (`data*`): + +[source,terminal] +---- +coreos.inst.save_partindex=’data*’ +---- + +[id="installation-user-infra-machines-advanced_ignition_{context}"] +== Identifying Ignition configs +There are different ways to manage Ignition config files when you do +bare metal installations. To begin with, there are two different types +of Ignition configs you can provide during {op-system-first} installation and two +different reasons for providing them: + +* **Live install Ignition config**: This Ignition config should be rarely +used and is one you create manually from scratch. It is passed to the live +install medium and it is run immediately upon booting that live medium to do setup +tasks BEFORE the process that actually installs the {op-system-first} system to disk. +This should only be used for performing tasks that need to be done once and +not applied again later, such as some advanced partitioning that cannot be done using MachineConfigs. There are currently no officially supported procedures for using Ignition configs in this way. ++ +For PXE or ISO boots, you can create the Ignition config +and APPEND the `ignition.config.url=` option to identify the location of +the Ignition config. You also need to append `ignition.firstboot ignition.platform.id=metal` +or the `ignition.config.url` option will be ignored. + +* **Permanent install Ignition config**: Every bare metal {op-system-first} installation +needs to pass one of the Ignition config files generated by `openshift-installer`, +including `bootstrap.ign`, `master.ign` and `worker.ign`, to the carry out the +installation. It is not recommended to modify these files. ++ +For PXE installs, you pass the Ignition configs on the APPEND line using the +`coreos.inst.ignition_url=` option. For ISO installs, after the ISO boots to +the shell prompt, you identify the Ignition config on the `coreos-installer` +command line with the `--ignition-url=` option. ++ +Instead of passing the location of an Ignition config via a kernel or +command-line option, you can embed an Ignition config into the ISO +installer image. This allows you to do bare metal installs with the ISO, +without requiring access to an HTTP server. See “Embedding an Ignition +config in the {op-system-first} ISO” for details. + +[id="installation-user-infra-machines-advanced_embedignition_{context}"] +=== Embedding an Ignition config in the {op-system-first} ISO +The following procedures describe how to embed an Ignition config into +the ISO so it is applied when the new installation first boots from disk. + +To embed an Ignition config named `worker.ign` into an ISO image +(for example rhcos--live.x86_64.iso), copy the image to +a local directory, then run the coreos-installer container with +that directory mounted, as follows: + +.Procedure + +. Get the {op-system-first} ISO and Ignition config file and copy them into an accessible directory, such as `/mnt`. ++ +[source,terminal] +---- +# cp rhcos--live.x86_64.iso bootstrap.ign /mnt/ +# chmod 644 /mnt/rhcos--live.x86_64.iso +---- + +. Run the following command to run `coreos-installer` from a container to embed the +Ignition config into the ISO: ++ +[source,terminal] +---- +# podman run -it -v /mnt:/mnt:z quay.io/coreos/coreos-installer:release \ + iso ignition embed --force -i /mnt/bootstrap.ign /mnt/rhcos.test.iso +---- + +You can now use that ISO to install {op-system-first} with the included Ignition config +without needing to pull the Ignition config from an HTTP server. + +To show the contents of the embedded Ignition config and direct it into a file, run: + +[source,terminal] +---- +# podman run -it -v /mnt:/mnt:z quay.io/coreos/coreos-installer:release \ + iso ignition show /mnt/rhcos.test.iso > mybootstrap.ign +# diff -s bootstrap.ign mybootstrap.ign +Files bootstrap.ign and mybootstrap.ign are identical +---- + +To remove the Ignition config and return the ISO to its pristine state (so +you can reuse it), run: + +[source,terminal] +---- +# podman run -it -v /mnt:/mnt:z quay.io/coreos/coreos-installer:release \ + iso ignition remove /mnt/rhcos.test.iso +---- + +You can now embed another Ignition config into the ISO or use the ISO in its +pristine state.