Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -73,14 +73,33 @@ Node(s) can become stuck when removing all Nodes from a cluster (scaling NodePoo

Several conditions can prevent Node(s) from being drained successfully:

- The hosted cluster contains `PodDisruptionBudgets` that require at least
- The hosted cluster contains pods that use `PersistentVolumes``
- The hosted cluster contains `PodDisruptionBudgets` that require at least one healthy pod, preventing eviction when there are no other nodes to reschedule onto.
- The hosted cluster contains pods that use `PersistentVolumes` that cannot be detached from the node.

!!! important

This is expected behavior. When all nodes are removed simultaneously, pods protected by PodDisruptionBudgets cannot be evicted because the PDB constraints cannot be satisfied with no remaining nodes. As a result, the drain operation blocks indefinitely. Configure `nodeDrainTimeout` to ensure nodes are eventually removed after a bounded period.

#### Prevention

To prevent Nodes from becoming stuck when scaling down, set the `.spec.nodeDrainTimeout` and `.spec.nodeVolumeDetachTimeout` in the NodePool CR to a value greater than `0s`.

This forces Nodes to be removed once the timeout specified in the field has been reached, regardless of whether the node can be drained or the volumes can be detached successfully.

```
apiVersion: hypershift.openshift.io/v1beta1
kind: NodePool
metadata:
name: example
namespace: clusters
spec:
nodeDrainTimeout: 30m
nodeVolumeDetachTimeout: 10m
# ...other fields...
```
Comment thread
coderabbitai[bot] marked this conversation as resolved.

!!! note
See the [Hypershift API reference page](../../reference/api.md) for more details.

See the [HyperShift API reference page](../../reference/api.md) for more details on these fields.

For an alternative approach that skips draining entirely via machine annotations, see [Scaling down data plane to Zero](scale-to-zero-dataplane.md).

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[nit] These two lines will render as a single dense paragraph in MkDocs since there is no blank line between them. Add a blank indented line between them for better readability.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@bryan-cox I have fixed it. Also aligned the !!! note block format to match the !!! important blocks in the same file.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

There is a issue here if we are adding blank line then hyperlinks are not showing in the doc preview see https://github.com/PoornimaSingour/hypershift/blob/f82366f09df63b6b2a990a8a7854a8e07c963331/docs/content/how-to/automated-machine-management/nodepool-lifecycle.md commit.

The issue is how GitHub renders this — GitHub doesn't understand MkDocs !!! note admonition syntax.
It treats the indented content as a code block or plain text, so the markdown links inside don't render as hyperlinks.

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Seems like it will look like this in the Github but in upstream it will come in hyperlinks. Changes are done

25 changes: 22 additions & 3 deletions docs/content/reference/aggregated-docs.md

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading