New MCO post-install tasks section

_topic_map.yml

@@ -311,6 +311,8 @@ Name: Post-installation configuration
 Dir: post_installation_configuration
 Distros: openshift-origin,openshift-enterprise,openshift-webscale
 Topics:
+- Name: Machine configuration tasks
+  File: machine-configuration-tasks
 - Name: Cluster tasks
   File: cluster-tasks
 - Name: Node tasks

modules/checking-mco-status.adoc

@@ -0,0 +1,112 @@
+// Module included in the following assemblies:
+//
+// * post_installation_configuration/machine-configuration-tasks.adoc
+
+[id="checking-mco-status_{context}"]
+= Checking Machine Config Pool status
+
+To see the status of the Machine Config Operator, its sub-components,
+and the resources it manages, use the following `oc` commands:
+
+.Procedure
+. To see the number of MCO-managed nodes available on your cluster for each pool, type:
++
+[source,terminal]
+----
+$ oc get machineconfigpool
+NAME      CONFIG                  UPDATED  UPDATING   DEGRADED  MACHINECOUNT  READYMACHINECOUNT  UPDATEDMACHINECOUNT DEGRADEDMACHINECOUNT  AGE
+master    rendered-master-dd…     True     False      False     3             3                  3                                0                     4h42m
+worker    rendered-worker-fde…    True     False      False     3             3                  3                                0                     4h42m
+----
++
+In the previous output, there are three master and three worker nodes. All machines are updated
+and none are currently updating. Because all nodes are Updated and Ready and none are Degraded,
+you can ell that there are no issues.
+
+. To see each existing `machineconfig`, type:
++
+[source,terminal]
+----
+$ oc get machineconfigs
+NAME                             GENERATEDBYCONTROLLER          IGNITIONVERSION  AGE
+00-master                        2c9371fbb673b97a6fe8b1c52...   3.1.0            5h18m
+00-worker                        2c9371fbb673b97a6fe8b1c52...   3.1.0            5h18m
+01-master-container-runtime      2c9371fbb673b97a6fe8b1c52...   3.1.0            5h18m
+01-master-kubelet                2c9371fbb673b97a6fe8b1c52…     3.1.0            5h18m
+...
+rendered-master-dde...           2c9371fbb673b97a6fe8b1c52...   3.1.0            5h18m
+rendered-worker-fde...           2c9371fbb673b97a6fe8b1c52...   3.1.0            5h18m
+----
++
+Note that the `machineconfigs` listed as `rendered` are not meant to be
+changed or deleted. Expect them to be hidden at some point in the future.
+
+. Check the status of worker (or change to master) to see the status of that pool of nodes:
++
+[source,terminal]
+----
+$ oc describe mcp worker
+...
+  Degraded Machine Count:     0
+  Machine Count:              3
+  Observed Generation:        2
+  Ready Machine Count:        3
+  Unavailable Machine Count:  0
+  Updated Machine Count:      3
+Events:                       <none>
+----
+
+. You can view the contents of a particular machineconfig (in this case,
+`01-master-kubelet`). The trimmed output from the following `oc describe`
+command shows that this `machineconfig` contains both configuration files
+(`cloud.conf` and `kubelet.conf`) and a systemd service
+(Kubernetes Kubelet):
++
+[source,terminal]
+----
+$ oc describe machineconfigs 01-master-kubelet
+Name:         01-master-kubelet
+...
+Spec:
+  Config:
+    Ignition:
+      Version:  3.1.0
+    Storage:
+      Files:
+        Contents:
+          Source:   data:,
+        Mode:       420
+        Overwrite:  true
+        Path:       /etc/kubernetes/cloud.conf
+        Contents:
+          Source:   data:,kind%3A%20KubeletConfiguration%0AapiVersion%3A%20kubelet.config.k8s.io%2Fv1beta1%0Aauthentication%3A%0A%20%20x509%3A%0A%20%20%20%20clientCAFile%3A%20%2Fetc%2Fkubernetes%2Fkubelet-ca.crt%0A%20%20anonymous...
+        Mode:       420
+        Overwrite:  true
+        Path:       /etc/kubernetes/kubelet.conf
+    Systemd:
+      Units:
+        Contents:  [Unit]
+Description=Kubernetes Kubelet
+Wants=rpc-statd.service network-online.target crio.service
+After=network-online.target crio.service
+
+ExecStart=/usr/bin/hyperkube \
+    kubelet \
+      --config=/etc/kubernetes/kubelet.conf \ ...
+----
+
+If something goes wrong with a machineconfig that you apply, you can always
+back out that change. For example, if you had run `oc create -f ./myconfig.yaml`
+to apply a machineconfig, you could remove that machineconfig by typing:
++
+[source,terminal]
+----
+$ oc delete -f ./myconfig.yaml
+----
++
+If that was the only problem, the nodes in the affected pool should return to a non-degraded state.
+This actually causes the rendered configuration to roll back to its previously rendered state.
+
+If you add your own MachineConfigs to your cluster, you can use the commands
+shown in the previous example to check their status and the related status of
+the pool to which they are applied.

modules/create-a-containerruntimeconfig-crd.adoc

@@ -0,0 +1,115 @@
+// Module included in the following assemblies:
+//
+// * post_installation_configuration/machine-configuration-tasks.adoc
+
+[id="create-a-containerruntimeconfig_{context}"]
+
+= Creating a ContainerRuntime CR to edit CRI-O parameters
+The ContainerRuntimeConfig custom resource definition (CRD) provides a
+structured way of changing settings associated with the {product-title}
+CRI-O runtime. Using a ContainerRuntimeConfig custom resource (CR),
+you select the configuration values you want and the MCO handles rebuilding
+the `crio.conf` and `storage.conf` configuration files.
+
+Parameters you can set in a ContainerRuntimeConfig CR include:
+
+* **PIDs limit**: Sets the maximum number of processes allowed in a container.
+By default, the limit is set to 1024 (`pids_limit = 1024`).
+* **Log level**: Sets the level of verbosity for log messages. The default is
+`info` (`log_level = info`). Other options include `fatal`, `panic`, `error`,
+`warn`, `debug`, and `trace`.
+* **Overlay size**: Sets the maxim size of a container image. The default is
+10 GB.
+* **Maximum log size**: Sets the maximum size allowed for the container
+log file. The default maximum log size is unlimited (`log_size_max = -1`).
+If it is set to a positive number, it must be at least 8192
+to not be smaller than `conmon`'s read buffer. Conmon is a program that
+monitors communications between a container manager (such as Podman or CRI-O)
+and the OCI runtime (such as runc or crun) for a single container.
+
+The following procedure describes how to change CRI-O settings using the
+ContainerRuntimeConfig CR.
+
+.Procedure
+
+. To raise the `pidsLimit` to 2048, set the `logLevel` to `debug`, and
+set the `overlaySize` to 8 GB, create a CR file
+(for example, `overlay-size.yaml`) that contains that setting:
++
+[source,yaml]
+----
+$ cat << EOF > /tmp/overlay-size.yaml
+apiVersion: machineconfiguration.openshift.io/v1
+kind: ContainerRuntimeConfig
+metadata:
+ name: overlay-size
+spec:
+ machineConfigPoolSelector:
+   matchLabels:
+     custom-crio: overlay-size
+ containerRuntimeConfig:
+   pidsLimit: 2048
+   logLevel: debug
+   overlaySize: 8G
+EOF
+----
+
+. To apply the ContainerRuntimeConfig settings, run:
++
+[source,terminal]
+----
+$ oc create -f /tmp/overlay-size
+----
+
+. To verify that the settings wer applied, run:
++
+[source,terminal]
+----
+$ oc get ContainerRuntimeConfig
+NAME           AGE
+overlay-size   3m19s
+
+----
+
+. To edit a pool of machines, such as `worker`, run the following
+command to open a MachineConfigPool:
++
+[source,terminal]
+----
+$ oc edit machineconfigpool worker
+----
+
+. Check that a new containerruntime object has appeared under the machineconfigs:
++
+[source,terminal]
+----
+$ oc get machineconfigs | grep containerrun
+99-worker-generated-containerruntime   2c9371fbb673b97a6fe8b1c52691999ed3a1bfc2  3.1.0  31s
+----
+. Monitor the Machine Config Pool as the changes are rolled into the machines until all are shown as ready:
++
+[source,terminal]
+----
+$ oc get mcp worker
+NAME    CONFIG               UPDATED  UPDATING  DEGRADED  MACHINECOUNT  READYMACHINECOUNT  UPDATEDMACHINECOUNT  DEGRADEDMACHINECOUNT  AGE
+worker  rendered-worker-169  False    True      False     3             1                  1                    0                     9h
+----
+
+. Open an `oc debug` session to a worker node and run `chroot /host`.
+
+. Verify the changes by running:
++
+[source,terminal]
+----
+$ crio config | egrep 'log_level|pids_limit'
+pids_limit = 2048
+log_level = "debug"
+$ head -n 7 /etc/containers/storage.conf
+[storage]
+  driver = "overlay"
+  runroot = "/var/run/containers/storage"
+  graphroot = "/var/lib/containers/storage"
+  [storage.options]
+    additionalimagestores = []
+    size = "8G"
+----

modules/create-a-kubeletconfig-crd-to-edit-kubelet-parameters.adoc

@@ -2,6 +2,7 @@
 //
 // * scalability_and_performance/recommended-host-practices.adoc
 // * post_installation_configuration/node-tasks.adoc
+// * post_installation_configuration/machine-configuration-tasks.adoc
 
 [id="create-a-kubeletconfig-crd-to-edit-kubelet-parameters_{context}"]
 = Creating a KubeletConfig CRD to edit kubelet parameters
@@ -91,13 +92,6 @@ spec:
     kubeAPIBurst: <burst_rate>
     kubeAPIQPS: <QPS>
 ----
-+
-.. Run:
-+
-[source,terminal]
-----
-$ oc label machineconfigpool worker custom-kubelet=large-pods
-----
 
 .. Run:
 +

modules/installation-special-config-crony.adoc

@@ -1,6 +1,7 @@
 // Module included in the following assemblies:
 //
 // * installing/install_config/installing-customizing.adoc
+// * post_installation_configuration/machine-configuration-tasks.adoc
 
 [id="installation-special-config-crony_{context}"]
 = Configuring chrony time service
@@ -68,10 +69,12 @@ EOF
 
 . Make a backup copy of the configuration file.
 
-. If the cluster is not up yet, generate manifest files, add this file to the `openshift`
-directory, then continue to create the cluster.
-
-. If the cluster is already running, apply the file as follows:
+. Apply the configuration in one of two ways:
++
+* If the cluster is not up yet, generate manifest files, add this file to the `openshift`
+directory, and then continue to create the cluster.
++
+* If the cluster is already running, apply the file as follows:
 +
 [source,terminal]
 ----

modules/machine-config-operator.adoc

@@ -1,14 +1,15 @@
 // Module included in the following assemblies:
 //
-// *  operators/operator-reference.adoc
+// * operators/operator-reference.adoc
+// * post_installation_configuration/machine-configuration-tasks.adoc
 
 [id="machine-config-operator_{context}"]
 = Machine Config Operator
 
 [discrete]
 == Purpose
 
-The Machine Congig Operator manages and applies configuration and updates of the
+The Machine Config Operator manages and applies configuration and updates of the
 base operating system and container runtime, including everything between the
 kernel and kubelet.