diff --git a/docs/user/customization.md b/docs/user/customization.md index c6659757458..9ef68a0551d 100644 --- a/docs/user/customization.md +++ b/docs/user/customization.md @@ -27,6 +27,175 @@ The `manifests` target will render the manifest templates and output the result The `manifest-templates` target will output the unrendered manifest templates into the asset directory. This allows modification to the templates before they have been rendered, which may be useful to users who wish to reuse the templates between cluster deployments. +### Install Time Customization for Machine Configuration + +**IMPORTANT**: + +- These customizations require using the `manifests` target that does not provide compatibility guarantees, for more information [check here](versioning.md#versioning). +- This can affect upgradability of your cluster as the `machine-config-operator` can mark clusters tainted when user defined [MachineConfig][machine-config] objects are present in the cluster. + +In most cases, user applications should be run on the cluster via Kubernetes workload objects (e.g. DaemonSet, Deployment, etc). For example, DaemonSets are the most stable way to run a logging agent on all hosts. However, there may be some cases where these workloads need to be executed prior to the node joining the Kubernetes cluster. For example, a compliance mandate like "the user must run auditing tools as soon as the operating system comes up" might require a custom systemd unit for an auditing container in the Ignition config for some or all nodes. + +The configuration of machines in OpenShift is controlled using `MachineConfig` objects and what configuration is applied to a machine in the OpenShift cluster is based on the [MachineConfigPool][machine-config-pool] objects. To allow customization of machine configuration which is not possible as Day 2 operation, the installer allows users to bring their own custom `MachineConfig` objects. + +1. `openshift-install --dir $INSTALL_DIR create manifests` + +2. Copy files with `MachineConfig` objects to `$INSTALL_DIR/openshift/` directory. + + These custom `MachineConfig` objects are black boxes to the installer and the installer only plays the role of `oc create -f ` early enough into cluster bootstrap to make sure the configuration is used by the [MachineConfigOperator][machine-config-operator]. + +3. `openshift-install --dir $INSTALL_DIR create cluster` + +#### Control plane with no Taints + +All control plane nodes by default register with a taint `node-role.kubernetes.io/master=:NoSchedule` making them unschedulable by most normal workloads. An installation that requires the control plane to boot without that taint can push a custom `MachineConfig` object with a `kubelet.service` that doesn't include the taint. + +For example: + +1. Run `manifests` target to create all the manifests. + + ```console + $ mkdir no-taint-cluster + + $ cp aws-install-config.yaml no-taint-cluster/install-config.yaml + + $ openshift-install --dir no-taint-cluster create manifests + INFO Consuming "Install Config" from target directory + + $ ls -l no-taint-cluster/** + no-taint-cluster/manifests: + total 68 + -rw-r--r--. 1 xxxxx xxxxx 169 Feb 28 10:54 04-openshift-machine-config-operator.yaml + -rw-r--r--. 1 xxxxx xxxxx 1589 Feb 28 10:54 cluster-config.yaml + -rw-r--r--. 1 xxxxx xxxxx 149 Feb 28 10:54 cluster-dns-02-config.yml + -rw-r--r--. 1 xxxxx xxxxx 243 Feb 28 10:54 cluster-infrastructure-02-config.yml + -rw-r--r--. 1 xxxxx xxxxx 154 Feb 28 10:54 cluster-ingress-02-config.yml + -rw-r--r--. 1 xxxxx xxxxx 557 Feb 28 10:54 cluster-network-01-crd.yml + -rw-r--r--. 1 xxxxx xxxxx 327 Feb 28 10:54 cluster-network-02-config.yml + -rw-r--r--. 1 xxxxx xxxxx 264 Feb 28 10:54 cvo-overrides.yaml + -rw-r--r--. 1 xxxxx xxxxx 275 Feb 28 10:54 etcd-service.yaml + -rw-r--r--. 1 xxxxx xxxxx 283 Feb 28 10:54 host-etcd-service-endpoints.yaml + -rw-r--r--. 1 xxxxx xxxxx 268 Feb 28 10:54 host-etcd-service.yaml + -rw-r--r--. 1 xxxxx xxxxx 118 Feb 28 10:54 kube-cloud-config.yaml + -rw-r--r--. 1 xxxxx xxxxx 1299 Feb 28 10:54 kube-system-configmap-etcd-serving-ca.yaml + -rw-r--r--. 1 xxxxx xxxxx 1304 Feb 28 10:54 kube-system-configmap-root-ca.yaml + -rw-r--r--. 1 xxxxx xxxxx 3877 Feb 28 10:54 kube-system-secret-etcd-client.yaml + -rw-r--r--. 1 xxxxx xxxxx 4030 Feb 28 10:54 machine-config-server-tls-secret.yaml + -rw-r--r--. 1 xxxxx xxxxx 856 Feb 28 10:54 pull.json + + no-taint-cluster/openshift: + total 28 + -rw-r--r--. 1 xxxxx xxxxx 293 Feb 28 10:54 99_binding-discovery.yaml + -rw-r--r--. 1 xxxxx xxxxx 181 Feb 28 10:54 99_kubeadmin-password-secret.yaml + -rw-r--r--. 1 xxxxx xxxxx 330 Feb 28 10:54 99_openshift-cluster-api_cluster.yaml + -rw-r--r--. 1 xxxxx xxxxx 1015 Feb 28 10:54 99_openshift-cluster-api_master-machines-0.yaml + -rw-r--r--. 1 xxxxx xxxxx 2655 Feb 28 10:54 99_openshift-cluster-api_master-user-data-secret.yaml + -rw-r--r--. 1 xxxxx xxxxx 1750 Feb 28 10:54 99_openshift-cluster-api_worker-machineset.yaml + -rw-r--r--. 1 xxxxx xxxxx 2655 Feb 28 10:54 99_openshift-cluster-api_worker-user-data-secret.yaml + ``` + +2. Create a `MachineConfig` that includes `kubelet.service` that has no taints. + + ```sh + cat > no-taint-cluster/openshift/99-master-kubelet-no-taint.yaml <