Skip to content

OTA-1174: upgrade status: add --detailed=nodes that shows all nodes #1735

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
openshift-merge-bot merged 2 commits into from
Apr 18, 2024
Merged

OTA-1174: upgrade status: add --detailed=nodes that shows all nodes #1735

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
17c3c1f
`upgrade status`: add `--detailed=nodes` that shows all nodes
petr-muller Apr 17, 2024
38cd6c9
`upgrade status`: test `--detailed=all` in examples integration test
petr-muller Apr 17, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
41 changes: 41 additions & 0 deletions pkg/cli/admin/upgrade/status/examples/4.14.1-degraded.detailed-output
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
An update is in progress for 1h58m50s: Unable to apply 4.14.1: wait has exceeded 40 minutes for these operators: etcd, kube-apiserver

Failing=True:

Reason: ClusterOperatorsDegraded
Message: Cluster operators etcd, kube-apiserver are degraded


= Control Plane =
Assessment: Progressing
Completion: 97%
Duration: 1h58m50s
Operator Status: 33 Total, 32 Available, 1 Progressing, 4 Degraded

Control Plane Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ip-10-0-30-217.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ?
ip-10-0-53-40.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ?
ip-10-0-92-180.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ?

= Worker Upgrade =

= Worker Pool =
Worker Pool: worker
Assessment: Pending
Completion: 0%
Worker Status: 3 Total, 3 Available, 0 Progressing, 3 Outdated, 0 Draining, 0 Excluded, 0 Degraded

Worker Pool Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ip-10-0-20-162.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ?
ip-10-0-4-159.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ?
ip-10-0-99-40.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ?

= Update Health =
SINCE LEVEL IMPACT MESSAGE
58m18s Error API Availability Cluster Operator kube-apiserver is degraded | NodeController_MasterNodesReady: NodeControllerDegraded: The master nodes not ready: node "ip-10-0-12-74.ec2.internal" not ready since 2023-11-03 16:28:43 +0000 UTC because KubeletNotReady (container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)
58m18s Error API Availability Cluster Operator kube-controller-manager is degraded | NodeController_MasterNodesReady: NodeControllerDegraded: The master nodes not ready: node "ip-10-0-12-74.ec2.internal" not ready since 2023-11-03 16:28:43 +0000 UTC because KubeletNotReady (container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)
58m18s Error API Availability Cluster Operator kube-scheduler is degraded | NodeController_MasterNodesReady: NodeControllerDegraded: The master nodes not ready: node "ip-10-0-12-74.ec2.internal" not ready since 2023-11-03 16:28:43 +0000 UTC because KubeletNotReady (container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)
58m38s Error API Availability Cluster Operator etcd is degraded | EtcdEndpoints_ErrorUpdatingEtcdEndpoints::EtcdMembers_UnhealthyMembers::NodeController_MasterNodesReady: EtcdEndpointsDegraded: EtcdEndpointsController can't evaluate whether quorum is safe: etcd cluster has quorum of 2 and 2 healthy members which is not fault tolerant: [{Member:ID:12895393557789359222 name:"ip-10-0-73-118.ec2.internal" peerURLs:"https://10.0.73.118:2380" clientURLs:"https://10.0.73.118:2379" Healthy:true Took:1.725492ms Error:<nil>} {Member:ID:13608765340770574953 name:"ip-10-0-0-60.ec2.internal" peerURLs:"https://10.0.0.60:2380" clientURLs:"https://10.0.0.60:2379" Healthy:true Took:1.542919ms Error:<nil>} {Member:ID:18044478200504924924 name:"ip-10-0-12-74.ec2.internal" peerURLs:"https://10.0.12.74:2380" clientURLs:"https://10.0.12.74:2379" Healthy:false Took: Error:create client failure: failed to make etcd client for endpoints [https://10.0.12.74:2379]: context deadline exceeded}] // EtcdMembersDegraded: 2 of 3 members are available, ip-10-0-12-74.ec2.internal is unhealthy // NodeControllerDegraded: The master nodes not ready: node "ip-10-0-12-74.ec2.internal" not ready since 2023-11-03 16:28:43 +0000 UTC because KubeletNotReady (container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?)
1h0m17s Error API Availability Cluster Operator control-plane-machine-set is unavailable | UnavailableReplicas: Missing 1 available replica(s)
31 changes: 31 additions & 0 deletions pkg/cli/admin/upgrade/status/examples/4.14.1-paused-worker-pool.detailed-output
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
An update is in progress for 6s: Working towards 4.14.1: 139 of 859 done (16% complete), waiting on kube-scheduler

= Control Plane =
Assessment: Progressing
Completion: 12%
Duration: 6s
Operator Status: 33 Total, 33 Available, 0 Progressing, 0 Degraded

Control Plane Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ip-10-0-30-217.us-east-2.compute.internal Outdated Pending 4.14.0 ?
ip-10-0-53-40.us-east-2.compute.internal Outdated Pending 4.14.0 ?
ip-10-0-92-180.us-east-2.compute.internal Outdated Pending 4.14.0 ?

= Worker Upgrade =

= Worker Pool =
Worker Pool: worker
Assessment: Excluded
Completion: 0%
Worker Status: 3 Total, 3 Available, 0 Progressing, 3 Outdated, 0 Draining, 3 Excluded, 0 Degraded

Worker Pool Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ip-10-0-20-162.us-east-2.compute.internal Excluded Paused 4.14.0 -
ip-10-0-4-159.us-east-2.compute.internal Excluded Paused 4.14.0 -
ip-10-0-99-40.us-east-2.compute.internal Excluded Paused 4.14.0 -

= Update Health =
SINCE LEVEL IMPACT MESSAGE
- Warning Update Stalled Worker pool worker is paused | Outdated nodes in a paused pool will not be updated.
41 changes: 41 additions & 0 deletions ...in/upgrade/status/examples/4.14.1-workers-started-updating-multiple-pools.detailed-output
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,41 @@
An update is in progress for 14m4s: Working towards 4.14.1: 734 of 859 done (85% complete), waiting on machine-config

= Control Plane =
Assessment: Progressing
Completion: 97%
Duration: 14m4s
Operator Status: 33 Total, 32 Available, 1 Progressing, 0 Degraded

Control Plane Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ip-10-0-53-40.us-east-2.compute.internal Progressing Draining 4.14.0 +30m
ip-10-0-30-217.us-east-2.compute.internal Outdated Pending 4.14.0 ?
ip-10-0-92-180.us-east-2.compute.internal Outdated Pending 4.14.0 ?

= Worker Upgrade =

= Worker Pool =
Worker Pool: worker
Assessment: Progressing
Completion: 0%
Worker Status: 3 Total, 2 Available, 1 Progressing, 3 Outdated, 1 Draining, 0 Excluded, 0 Degraded

Worker Pool Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ip-10-0-4-159.us-east-2.compute.internal Progressing Draining 4.14.0 +30m
ip-10-0-20-162.us-east-2.compute.internal Outdated Pending 4.14.0 ?
ip-10-0-99-40.us-east-2.compute.internal Outdated Pending 4.14.0 ?

= Worker Pool =
Worker Pool: infra
Assessment: Progressing
Completion: 0%
Worker Status: 1 Total, 0 Available, 1 Progressing, 1 Outdated, 1 Draining, 0 Excluded, 0 Degraded

Worker Pool Node
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ip-10-0-4-159-infra.us-east-2.compute.internal Progressing Draining 4.14.0 +30m

= Update Health =
SINCE LEVEL IMPACT MESSAGE
14m4s Info None Upgrade is proceeding well
31 changes: 31 additions & 0 deletions pkg/cli/admin/upgrade/status/examples/4.15.0-ec2-b02-cos-not-annotated.detailed-output
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
An update is in progress for 52m56s: Working towards 4.15.0-ec.2: 357 of 1021 done (34% complete), waiting up to 40 minutes on cluster-api

= Control Plane =
Assessment: Progressing
Completion: 43%
Duration: 52m56s
Operator Status: 7 Total, 7 Available, 0 Progressing, 0 Degraded

Control Plane Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ip-10-0-30-217.us-east-2.compute.internal Outdated Pending 4.15.0-ec.1 ?
ip-10-0-53-40.us-east-2.compute.internal Outdated Pending 4.15.0-ec.1 ?
ip-10-0-92-180.us-east-2.compute.internal Outdated Pending 4.15.0-ec.1 ?

= Worker Upgrade =

= Worker Pool =
Worker Pool: worker
Assessment: Pending
Completion: 0%
Worker Status: 3 Total, 3 Available, 0 Progressing, 3 Outdated, 0 Draining, 0 Excluded, 0 Degraded

Worker Pool Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ip-10-0-20-162.us-east-2.compute.internal Outdated Pending 4.15.0-ec.1 ?
ip-10-0-4-159.us-east-2.compute.internal Outdated Pending 4.15.0-ec.1 ?
ip-10-0-99-40.us-east-2.compute.internal Outdated Pending 4.15.0-ec.1 ?

= Update Health =
SINCE LEVEL IMPACT MESSAGE
52m56s Info None Upgrade is proceeding well
31 changes: 31 additions & 0 deletions pkg/cli/admin/upgrade/status/examples/4.15.0-ec2-early.detailed-output
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
An update is in progress for 1m29s: Working towards 4.15.0-ec.2: 106 of 863 done (12% complete), waiting on etcd, kube-apiserver

= Control Plane =
Assessment: Progressing
Completion: 3%
Duration: 1m29s
Operator Status: 33 Total, 33 Available, 2 Progressing, 0 Degraded

Control Plane Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ip-10-0-30-217.us-east-2.compute.internal Outdated Pending 4.14.1 ?
ip-10-0-53-40.us-east-2.compute.internal Outdated Pending 4.14.1 ?
ip-10-0-92-180.us-east-2.compute.internal Outdated Pending 4.14.1 ?

= Worker Upgrade =

= Worker Pool =
Worker Pool: worker
Assessment: Pending
Completion: 0%
Worker Status: 3 Total, 3 Available, 0 Progressing, 3 Outdated, 0 Draining, 0 Excluded, 0 Degraded

Worker Pool Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ip-10-0-20-162.us-east-2.compute.internal Outdated Pending 4.14.1 ?
ip-10-0-4-159.us-east-2.compute.internal Outdated Pending 4.14.1 ?
ip-10-0-99-40.us-east-2.compute.internal Outdated Pending 4.14.1 ?

= Update Health =
SINCE LEVEL IMPACT MESSAGE
1m29s Info None Upgrade is proceeding well
31 changes: 31 additions & 0 deletions pkg/cli/admin/upgrade/status/examples/4.15.0-ec2-unavailable-mco-20m.detailed-output
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
An update is in progress for 58m53s: Working towards 4.15.0-ec.2: 110 of 863 done (12% complete), waiting up to 40 minutes on etcd

= Control Plane =
Assessment: Progressing
Completion: 97%
Duration: 58m53s
Operator Status: 33 Total, 31 Available, 1 Progressing, 1 Degraded

Control Plane Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ip-10-0-30-217.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ?
ip-10-0-53-40.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ?
ip-10-0-92-180.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ?

= Worker Upgrade =

= Worker Pool =
Worker Pool: worker
Assessment: Pending
Completion: 0%
Worker Status: 3 Total, 3 Available, 0 Progressing, 3 Outdated, 0 Draining, 0 Excluded, 0 Degraded

Worker Pool Nodes
NAME ASSESSMENT PHASE VERSION EST MESSAGE
ip-10-0-20-162.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ?
ip-10-0-4-159.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ?
ip-10-0-99-40.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ?

= Update Health =
SINCE LEVEL IMPACT MESSAGE
20m24s Error API Availability Cluster Operator machine-config is unavailable | MachineConfigControllerFailed: Cluster not available for [{operator 4.14.0-rc.3}]: ControllerConfig.machineconfiguration.openshift.io "machine-config-controller" is invalid: [status.controllerCertificates[0].notAfter: Required value, status.controllerCertificates[0].notBefore: Required value, status.controllerCertificates[1].notAfter: Required value, status.controllerCertificates[1].notBefore: Required value, status.controllerCertificates[2].notAfter: Required value, status.controllerCertificates[2].notBefore: Required value, status.controllerCertificates[3].notAfter: Required value, status.controllerCertificates[3].notBefore: Required value, status.controllerCertificates[4].notAfter: Required value, status.controllerCertificates[4].notBefore: Required value, status.controllerCertificates[5].notAfter: Required value, status.controllerCertificates[5].notBefore: Required value, status.controllerCertificates[6].notAfter: Required value, status.controllerCertificates[6].notBefore: Required value, status.controllerCertificates[7].notAfter: Required value, status.controllerCertificates[7].notBefore: Required value, status.controllerCertificates[8].notAfter: Required value, status.controllerCertificates[8].notBefore: Required value, status.controllerCertificates[9].notAfter: Required value, status.controllerCertificates[9].notBefore: Required value, <nil>: Invalid value: "null": some validation rules were not checked because the object was invalid; correct the existing errors to complete validation]
Loading