Rancher Integration: Upgrades in Rancher

[davidnuzik]

Standalone upgrade controller integration and integration with Rancher.
Base off of rancher/system-upgrade-controller framework.

Config of nodes: #43 (to be done later)

[MonzElmasry]

RKE2, SUC, rancher integration design
to be able to import rke2 cluster into rancher, and use SUC to upgrade rke2 cluster nodes.

Step 1:
Create an upgrade script for RKE2: (Done)

• create rke2-upgrade repo

• need rke2 installation script

• rke2 upgrade script:

  • search for rke2 binary
  • replace new binary, place it in the old binary path
  • kill rke2 process, (systemd will restart it)

• write example plan file

• set up channelserver for rke2

  • to be able to have channel: instead of version in plan file
  • create channel.yaml in rke2 repo

Step 2:
In rancher import cluster:

• detect if it's an RKE2 cluster
  .node.status.nodeInfo.containerRuntimeVersion
• set the driver for rke2 in rancher
• check cluster annotation (in cluster provisioner)
• edit cluster nodes (cluster.yaml file)
• expose upgrade functionality of rke2

Step 3:
In rancher RKE2 upgrade:

• check suc version and update it to latest
• check rke2 version through kubeconfig file
• check rke2 releases from channel server
• expose drain/ concurrency flags in ui
• create master/agent plan
• apply plans, check if deployed,
• check plan status from suc

changes in repos other than rancher

• RKE2-upgrade

  • Create rke2-upgrade

• rke2

  • State rke2 string in rke2 version
  • add channel.yaml

• channel server

  • point out to github.com/rancher/rke2/releases/ ... [stable/latest/testing/alpha?]

[rancher-max]

Brief Update - Tests are 90% complete

• All of the P1 and P0 test cases have been completed. The general rundown of this is as follows:

  • 3 server nodes, 3 agent nodes
  • Upgrade using Rancher UI and upgrade without it (directly apply system-upgrade-controller and plans)
  • Upgrade with different concurrency values
  • Upgrade with drain true/false
  • Ensure server nodes upgrade before agent nodes
  • Added nodes are upgraded
  • RBAC in Rancher
  • Upgrade hangs while server nodes are powered off, and continues when the node is powered back on
  • Single node cluster upgrade is successful
  • Additional parameters set upon initial install remain during upgrade

• What remains:

  • Greater scale upgrades (5 server, 15 agent for example with differing concurrency values)
  • Upgrading cluster with CIS mode enabled and ensuring it isn't lost upon upgrade

Currently blocked on finishing remainder of upgrade cases through Rancher UI by: rancher/rancher#28266

[rancher-max]

Validated rest of scenarios in master-d19f179..-head

• CIS mode upgrades keep CIS mode enabled and upgrade successfully
• Scaled clusters upgrade successfully with different master/worker concurrencies and drain on/off