Skip to content

Updating imagecontentsourcepolicy should not trigger a node drain #57440

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
mburke5678 merged 1 commit into from
Mar 24, 2023
Merged

Updating imagecontentsourcepolicy should not trigger a node drain #57440

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 2 additions & 10 deletions modules/troubleshooting-disabling-autoreboot-mco.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,17 +7,9 @@

When configuration changes are made by the Machine Config Operator (MCO), {op-system-first} must reboot for the changes to take effect. Whether the configuration change is automatic or manual, an {op-system} node reboots automatically unless it is paused.

[NOTE]
[NOTE]
====
The following modifications do not trigger a node reboot:

* When the MCO detects any of the following changes, it applies the update without draining or rebooting the node:

** Changes to the SSH key in the `spec.config.passwd.users.sshAuthorizedKeys` parameter of a machine config.
** Changes to the global pull secret or pull secret in the `openshift-config` namespace.
** Automatic rotation of the `/etc/kubernetes/kubelet-ca.crt` certificate authority (CA) by the Kubernetes API Server Operator.

* When the MCO detects changes to the `/etc/containers/registries.conf` file, such as adding or editing an `ImageDigestMirrorSet` or `ImageTagMirrorSet` object, it drains the corresponding nodes, applies the changes, and uncordons the nodes. The node drain does not happen when there is a mirror configuration change in the object, such as adding a new mirror to an existing registry by using an `ImageDigestMirrorSet` object. The MCO does restart the nodes if you add a new mirror to an existing registry by uisng an`ImageTagMirrorSet` object.
include::snippets/node-icsp-no-drain.adoc[]
====

To avoid unwanted disruptions, you can modify the machine config pool (MCP) to prevent automatic rebooting after the Operator makes changes to the machine config.
Expand Down
10 changes: 1 addition & 9 deletions modules/understanding-machine-config-operator.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,15 +48,7 @@ To prevent the nodes from automatically rebooting after machine configuration ch

Make sure the pools are unpaused when the CA certificate rotation happens. If the MCPs are paused, the MCO cannot push the newly rotated certificates to those nodes. This causes the cluster to become degraded and causes failure in multiple `oc` commands, including `oc debug`, `oc logs`, `oc exec`, and `oc attach`. You receive alerts in the Alerting UI of the {product-title} web console if an MCP is paused when the certificates are rotated.

The following modifications do not trigger a node reboot:

* When the MCO detects any of the following changes, it applies the update without draining or rebooting the node:

** Changes to the SSH key in the `spec.config.passwd.users.sshAuthorizedKeys` parameter of a machine config.
** Changes to the global pull secret or pull secret in the `openshift-config` namespace.
** Automatic rotation of the `/etc/kubernetes/kubelet-ca.crt` certificate authority (CA) by the Kubernetes API Server Operator.

* When the MCO detects changes to the `/etc/containers/registries.conf` file, such as adding or editing an `ImageDigestMirrorSet` or `ImageTagMirrorSet` object, it drains the corresponding nodes, applies the changes, and uncordons the nodes.The node drain does not happen when there is a mirror configuration change in the object, such as adding a new mirror to an existing registry by using an `ImageDigestMirrorSet` object. The MCO does restart the nodes if you add a new mirror to an existing registry by using an`ImageTagMirrorSet` object.
include::snippets/node-icsp-no-drain.adoc[]
====

There might be situations where the configuration on a node does not fully match what the currently-applied machine config specifies. This state is called _configuration drift_. The Machine Config Daemon (MCD) regularly checks the nodes for configuration drift. If the MCD detects configuration drift, the MCO marks the node `degraded` until an administrator corrects the node configuration. A degraded node is online and operational, but, it cannot be updated.
20 changes: 20 additions & 0 deletions snippets/node-icsp-no-drain.adoc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
// Text snippet included in the following modules:
//
// * modules/about-crio.adoc
// * modules/nodes-containers-using.adoc

:_content-type: SNIPPET

The following modifications do not trigger a node reboot:

* When the MCO detects any of the following changes, it applies the update without draining or rebooting the node:

** Changes to the SSH key in the `spec.config.passwd.users.sshAuthorizedKeys` parameter of a machine config.
** Changes to the global pull secret or pull secret in the `openshift-config` namespace.
** Automatic rotation of the `/etc/kubernetes/kubelet-ca.crt` certificate authority (CA) by the Kubernetes API Server Operator.

* When the MCO detects changes to the `/etc/containers/registries.conf` file, such as adding or editing an `ImageDigestMirrorSet` or `ImageTagMirrorSet` object, it drains the corresponding nodes, applies the changes, and uncordons the nodes.The node drain does not happen for the following changes:
Copy link

@rioliu-rh rioliu-rh Mar 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mburke5678 ImageDigestMirrorSet and ImageTagMirrorSet are new resource to us. we need to verify the behavior mentioned in the doc

Copy link

@sergiordlr sergiordlr Mar 23, 2023

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have verified the behavior today:

  • ImageDigestMirrorSet applies the mirrors with pull-from-mirror = "digest-only" and the drain operation is skipped.

  • ImageTagMirrorSet applies the mirrors with pull-from-mirror = "tag-only" and it performs a drain operation as expected.

** The addition of a registry with the `pull-from-mirror = "digest-only"` parameter set for each mirror.
** The addition of a mirror with the `pull-from-mirror = "digest-only"` parameter set in a registry.
** The addition of items to the `unqualified-search-registries` list.