Skip to content

Document entry-point for user-provided custom MachineConfig #1331

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-robot merged 1 commit into from
Mar 4, 2019
Merged

Document entry-point for user-provided custom MachineConfig #1331

Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
169 changes: 169 additions & 0 deletions docs/user/customization.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -27,6 +27,175 @@ The `manifests` target will render the manifest templates and output the result

The `manifest-templates` target will output the unrendered manifest templates into the asset directory. This allows modification to the templates before they have been rendered, which may be useful to users who wish to reuse the templates between cluster deployments.

### Install Time Customization for Machine Configuration

**IMPORTANT**:

- These customizations require using the `manifests` target that does not provide compatibility guarantees, for more information [check here](versioning.md#versioning).
- This can affect upgradability of your cluster as the `machine-config-operator` can mark clusters tainted when user defined [MachineConfig][machine-config] objects are present in the cluster.

In most cases, user applications should be run on the cluster via Kubernetes workload objects (e.g. DaemonSet, Deployment, etc). For example, DaemonSets are the most stable way to run a logging agent on all hosts. However, there may be some cases where these workloads need to be executed prior to the node joining the Kubernetes cluster. For example, a compliance mandate like "the user must run auditing tools as soon as the operating system comes up" might require a custom systemd unit for an auditing container in the Ignition config for some or all nodes.

The configuration of machines in OpenShift is controlled using `MachineConfig` objects and what configuration is applied to a machine in the OpenShift cluster is based on the [MachineConfigPool][machine-config-pool] objects. To allow customization of machine configuration which is not possible as Day 2 operation, the installer allows users to bring their own custom `MachineConfig` objects.

1. `openshift-install --dir $INSTALL_DIR create manifests`

2. Copy files with `MachineConfig` objects to `$INSTALL_DIR/openshift/` directory.

These custom `MachineConfig` objects are black boxes to the installer and the installer only plays the role of `oc create -f <custom-machine-config-object>` early enough into cluster bootstrap to make sure the configuration is used by the [MachineConfigOperator][machine-config-operator].

3. `openshift-install --dir $INSTALL_DIR create cluster`

#### Control plane with no Taints

All control plane nodes by default register with a taint `node-role.kubernetes.io/master=:NoSchedule` making them unschedulable by most normal workloads. An installation that requires the control plane to boot without that taint can push a custom `MachineConfig` object with a `kubelet.service` that doesn't include the taint.

For example:

1. Run `manifests` target to create all the manifests.

```console
$ mkdir no-taint-cluster

$ cp aws-install-config.yaml no-taint-cluster/install-config.yaml

$ openshift-install --dir no-taint-cluster create manifests
INFO Consuming "Install Config" from target directory

$ ls -l no-taint-cluster/**
no-taint-cluster/manifests:
total 68
-rw-r--r--. 1 xxxxx xxxxx 169 Feb 28 10:54 04-openshift-machine-config-operator.yaml
-rw-r--r--. 1 xxxxx xxxxx 1589 Feb 28 10:54 cluster-config.yaml
-rw-r--r--. 1 xxxxx xxxxx 149 Feb 28 10:54 cluster-dns-02-config.yml
-rw-r--r--. 1 xxxxx xxxxx 243 Feb 28 10:54 cluster-infrastructure-02-config.yml
-rw-r--r--. 1 xxxxx xxxxx 154 Feb 28 10:54 cluster-ingress-02-config.yml
-rw-r--r--. 1 xxxxx xxxxx 557 Feb 28 10:54 cluster-network-01-crd.yml
-rw-r--r--. 1 xxxxx xxxxx 327 Feb 28 10:54 cluster-network-02-config.yml
-rw-r--r--. 1 xxxxx xxxxx 264 Feb 28 10:54 cvo-overrides.yaml
-rw-r--r--. 1 xxxxx xxxxx 275 Feb 28 10:54 etcd-service.yaml
-rw-r--r--. 1 xxxxx xxxxx 283 Feb 28 10:54 host-etcd-service-endpoints.yaml
-rw-r--r--. 1 xxxxx xxxxx 268 Feb 28 10:54 host-etcd-service.yaml
-rw-r--r--. 1 xxxxx xxxxx 118 Feb 28 10:54 kube-cloud-config.yaml
-rw-r--r--. 1 xxxxx xxxxx 1299 Feb 28 10:54 kube-system-configmap-etcd-serving-ca.yaml
-rw-r--r--. 1 xxxxx xxxxx 1304 Feb 28 10:54 kube-system-configmap-root-ca.yaml
-rw-r--r--. 1 xxxxx xxxxx 3877 Feb 28 10:54 kube-system-secret-etcd-client.yaml
-rw-r--r--. 1 xxxxx xxxxx 4030 Feb 28 10:54 machine-config-server-tls-secret.yaml
-rw-r--r--. 1 xxxxx xxxxx 856 Feb 28 10:54 pull.json

no-taint-cluster/openshift:
total 28
-rw-r--r--. 1 xxxxx xxxxx 293 Feb 28 10:54 99_binding-discovery.yaml
-rw-r--r--. 1 xxxxx xxxxx 181 Feb 28 10:54 99_kubeadmin-password-secret.yaml
-rw-r--r--. 1 xxxxx xxxxx 330 Feb 28 10:54 99_openshift-cluster-api_cluster.yaml
-rw-r--r--. 1 xxxxx xxxxx 1015 Feb 28 10:54 99_openshift-cluster-api_master-machines-0.yaml
-rw-r--r--. 1 xxxxx xxxxx 2655 Feb 28 10:54 99_openshift-cluster-api_master-user-data-secret.yaml
-rw-r--r--. 1 xxxxx xxxxx 1750 Feb 28 10:54 99_openshift-cluster-api_worker-machineset.yaml
-rw-r--r--. 1 xxxxx xxxxx 2655 Feb 28 10:54 99_openshift-cluster-api_worker-user-data-secret.yaml
```

2. Create a `MachineConfig` that includes `kubelet.service` that has no taints.

```sh
cat > no-taint-cluster/openshift/99-master-kubelet-no-taint.yaml <<EOF
apiVersion: machineconfiguration.openshift.io/v1
kind: MachineConfig
metadata:
labels:
machineconfiguration.openshift.io/role: master
name: 02-master-kubelet
spec:
config:
ignition:
version: 2.2.0
systemd:
units:
- contents: |
[Unit]
Description=Kubernetes Kubelet
Wants=rpc-statd.service

[Service]
Type=notify
ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests
ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state
EnvironmentFile=-/etc/kubernetes/kubelet-workaround
EnvironmentFile=-/etc/kubernetes/kubelet-env

ExecStart=/usr/bin/hyperkube \
kubelet \
--config=/etc/kubernetes/kubelet.conf \
--bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
--rotate-certificates \
--kubeconfig=/var/lib/kubelet/kubeconfig \
--container-runtime=remote \
--container-runtime-endpoint=/var/run/crio/crio.sock \
--allow-privileged \
--node-labels=node-role.kubernetes.io/master \
--minimum-container-ttl-duration=6m0s \
--client-ca-file=/etc/kubernetes/ca.crt \
--cloud-provider=aws \
--volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec \
\
--anonymous-auth=false \

Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
enabled: true
name: kubelet.service
EOF
```

`machineconfiguration.openshift.io/role: master` label attaches this `MachineConfig` to the [master][master-machine-config-pool] `MachineConfigPool`. The [default][default-kubelet-service] configuration for the `kubelet.service` on libvirt includes the taint.

3. Run `cluster` target to create the cluster using the custom manifests.

```console
$ openshift-install --dir no-taint-cluster create cluster
INFO Consuming "Openshift Manifests" from target directory
INFO Consuming "Master Machines" from target directory
INFO Consuming "Common Manifests" from target directory
INFO Creating cluster...
INFO Waiting up to 30m0s for the Kubernetes API...
...
```

Check that no control plane nodes registered with taints:

```console
$ oc --config no-taint-cluster/auth/kubeconfig get nodes -ojson | jq '.items[] | select(.metadata.labels."node-role.kubernetes.io/master" == "") | .spec.taints'
Copy link
Contributor

@kikisdeliveryservice kikisdeliveryservice Feb 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a line about what this is doing (esp bc this is half hidden in my browser so the full command isn't visible without lots of scrolling).

Copy link
Contributor Author

@abhinavdahiya abhinavdahiya Mar 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Contributor

@kikisdeliveryservice kikisdeliveryservice Mar 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Perfecto!!

null
```

Check that the `02-master-kubelet` `MachineConfig` exists in the cluster:

```console
oc --config no-taint-cluster/auth/kubeconfig get machineconfigs
Copy link
Contributor

@kikisdeliveryservice kikisdeliveryservice Feb 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Add a line about what this is doing?

Copy link
Contributor Author

@abhinavdahiya abhinavdahiya Mar 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

done

Copy link
Contributor

@kikisdeliveryservice kikisdeliveryservice Mar 1, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

nice!

NAME GENERATEDBYCONTROLLER IGNITIONVERSION CREATED
00-master 3.11.0-744-g5b05d9d3-dirty 2.2.0 137m
00-master-ssh 3.11.0-744-g5b05d9d3-dirty 137m
00-worker 3.11.0-744-g5b05d9d3-dirty 2.2.0 137m
00-worker-ssh 3.11.0-744-g5b05d9d3-dirty 137m
01-master-container-runtime 3.11.0-744-g5b05d9d3-dirty 2.2.0 137m
01-master-kubelet 3.11.0-744-g5b05d9d3-dirty 2.2.0 137m
02-master-kubelet 2.2.0 137m
01-worker-container-runtime 3.11.0-744-g5b05d9d3-dirty 2.2.0 137m
01-worker-kubelet 3.11.0-744-g5b05d9d3-dirty 2.2.0 137m
99-master-3c81ffa3-3b8d-11e9-ac1e-52fdfc072182-registries 3.11.0-744-g5b05d9d3-dirty 133m
99-worker-3c83a226-3b8d-11e9-ac1e-52fdfc072182-registries 3.11.0-744-g5b05d9d3-dirty 133m
master-55491738d7cd1ad6c72891e77c35e024 3.11.0-744-g5b05d9d3-dirty 2.2.0 137m
worker-edab0895c59dba7a566f4b955d87d964 3.11.0-744-g5b05d9d3-dirty 2.2.0 137m
```

[default-kubelet-service]: https://github.com/openshift/machine-config-operator/blob/master/templates/master/01-master-kubelet/_base/units/kubelet.yaml
[machine-config-operator]: https://github.com/openshift/machine-config-operator#machine-config-operator
[machine-config-pool]: https://github.com/openshift/machine-config-operator/blob/master/docs/MachineConfigController.md#machinepool
[machine-config]: https://github.com/openshift/machine-config-operator/blob/master/docs/MachineConfiguration.md
[master-machine-config-pool]: https://github.com/openshift/machine-config-operator/blob/master/manifests/master.machineconfigpool.yaml

## OS Customization (unvalidated)

In rare circumstances, certain modifications to the bootstrap and other machines may be necessary. The installer provides the "ignition-configs" target, which allows arbitrary modification to the [Ignition Configs][ignition] used to boot these machines. Note that there is currently no validation on the modifications that are made, so it is possible that the changes will result in a non-functioning cluster.
Expand Down