-
Notifications
You must be signed in to change notification settings - Fork 474
OTA-1174: upgrade status: --details shows descriptions and references
#1690
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Closed
Closed
Changes from all commits
Commits
Show all changes
10 commits
Select commit
Hold shift + click to select a range
a0200a1
`upgrade status`: --details flag that shows reference and description
petr-muller 9956837
`upgrade status`: populate ref & description for CO insights
petr-muller 02f0566
`upgrade status`: keep linebreaks in description but respect indent
petr-muller 9f16602
`upgrade status`: ignore incomplete insights
petr-muller 77ecca9
`upgrade status`: minor fixes for node status
petr-muller 1d3922c
squash: regenerate detailed outputs
petr-muller a799988
squash: ellipsize long node names in node section
petr-muller 37f4409
`update status`: coalesce similar degraded node insights
petr-muller 40db17f
`upgrade status`: emit resources in detailed mode
petr-muller 6da69b9
`upgrade status`: separate detailed mode for nodes and health
petr-muller File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
81 changes: 81 additions & 0 deletions
81
pkg/cli/admin/upgrade/status/examples/4.14.1-degraded.detailed-output
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,81 @@ | ||
| An update is in progress for 1h58m50s: Unable to apply 4.14.1: wait has exceeded 40 minutes for these operators: etcd, kube-apiserver | ||
|
|
||
| Failing=True: | ||
|
|
||
| Reason: ClusterOperatorsDegraded | ||
| Message: Cluster operators etcd, kube-apiserver are degraded | ||
|
|
||
|
|
||
| = Control Plane = | ||
| Assessment: Progressing | ||
| Completion: 97% | ||
| Duration: 1h58m50s | ||
| Operator Status: 33 Total, 32 Available, 1 Progressing, 4 Degraded | ||
|
|
||
| Control Plane Nodes | ||
| NAME ASSESSMENT PHASE VERSION EST MESSAGE | ||
| ip-10-0-30-217.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ? | ||
| ip-10-0-53-40.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ? | ||
| ip-10-0-92-180.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ? | ||
|
|
||
| = Worker Upgrade = | ||
|
|
||
| = Worker Pool = | ||
| Worker Pool: worker | ||
| Assessment: Pending | ||
| Completion: 0% | ||
| Worker Status: 3 Total, 3 Available, 0 Progressing, 3 Outdated, 0 Draining, 0 Excluded, 0 Degraded | ||
|
|
||
| Worker Pool Nodes | ||
| NAME ASSESSMENT PHASE VERSION EST MESSAGE | ||
| ip-10-0-20-162.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ? | ||
| ip-10-0-4-159.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ? | ||
| ip-10-0-99-40.us-east-2.compute.internal Outdated Pending 4.14.0-rc.3 ? | ||
|
|
||
| = Update Health = | ||
| Message: Cluster Operator kube-apiserver is degraded (NodeController_MasterNodesReady) | ||
| Since: 58m18s | ||
| Level: Error | ||
| Impact: API Availability | ||
| Reference: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/ClusterOperatorDegraded.md | ||
| Resources: | ||
| ClusterOperators: kube-apiserver | ||
| Description: NodeControllerDegraded: The master nodes not ready: node "ip-10-0-12-74.ec2.internal" not ready since 2023-11-03 16:28:43 +0000 UTC because KubeletNotReady (container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?) | ||
|
|
||
| Message: Cluster Operator kube-controller-manager is degraded (NodeController_MasterNodesReady) | ||
| Since: 58m18s | ||
| Level: Error | ||
| Impact: API Availability | ||
| Reference: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/ClusterOperatorDegraded.md | ||
| Resources: | ||
| ClusterOperators: kube-controller-manager | ||
| Description: NodeControllerDegraded: The master nodes not ready: node "ip-10-0-12-74.ec2.internal" not ready since 2023-11-03 16:28:43 +0000 UTC because KubeletNotReady (container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?) | ||
|
|
||
| Message: Cluster Operator kube-scheduler is degraded (NodeController_MasterNodesReady) | ||
| Since: 58m18s | ||
| Level: Error | ||
| Impact: API Availability | ||
| Reference: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/ClusterOperatorDegraded.md | ||
| Resources: | ||
| ClusterOperators: kube-scheduler | ||
| Description: NodeControllerDegraded: The master nodes not ready: node "ip-10-0-12-74.ec2.internal" not ready since 2023-11-03 16:28:43 +0000 UTC because KubeletNotReady (container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?) | ||
|
|
||
| Message: Cluster Operator etcd is degraded (EtcdEndpoints_ErrorUpdatingEtcdEndpoints::EtcdMembers_UnhealthyMembers::NodeController_MasterNodesReady) | ||
| Since: 58m38s | ||
| Level: Error | ||
| Impact: API Availability | ||
| Reference: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/ClusterOperatorDegraded.md | ||
| Resources: | ||
| ClusterOperators: etcd | ||
| Description: EtcdEndpointsDegraded: EtcdEndpointsController can't evaluate whether quorum is safe: etcd cluster has quorum of 2 and 2 healthy members which is not fault tolerant: [{Member:ID:12895393557789359222 name:"ip-10-0-73-118.ec2.internal" peerURLs:"https://10.0.73.118:2380" clientURLs:"https://10.0.73.118:2379" Healthy:true Took:1.725492ms Error:<nil>} {Member:ID:13608765340770574953 name:"ip-10-0-0-60.ec2.internal" peerURLs:"https://10.0.0.60:2380" clientURLs:"https://10.0.0.60:2379" Healthy:true Took:1.542919ms Error:<nil>} {Member:ID:18044478200504924924 name:"ip-10-0-12-74.ec2.internal" peerURLs:"https://10.0.12.74:2380" clientURLs:"https://10.0.12.74:2379" Healthy:false Took: Error:create client failure: failed to make etcd client for endpoints [https://10.0.12.74:2379]: context deadline exceeded}] | ||
| , EtcdMembersDegraded: 2 of 3 members are available, ip-10-0-12-74.ec2.internal is unhealthy | ||
| , NodeControllerDegraded: The master nodes not ready: node "ip-10-0-12-74.ec2.internal" not ready since 2023-11-03 16:28:43 +0000 UTC because KubeletNotReady (container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: No CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?) | ||
|
|
||
| Message: Cluster Operator control-plane-machine-set is unavailable (UnavailableReplicas) | ||
| Since: 1h0m17s | ||
| Level: Error | ||
| Impact: API Availability | ||
| Reference: https://github.com/openshift/runbooks/blob/master/alerts/cluster-monitoring-operator/ClusterOperatorDown.md | ||
| Resources: | ||
| ClusterOperators: control-plane-machine-set | ||
| Description: Missing 1 available replica(s) |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
37 changes: 37 additions & 0 deletions
37
pkg/cli/admin/upgrade/status/examples/4.14.1-paused-worker-pool.detailed-output
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,37 @@ | ||
| An update is in progress for 6s: Working towards 4.14.1: 139 of 859 done (16% complete), waiting on kube-scheduler | ||
|
|
||
| = Control Plane = | ||
| Assessment: Progressing | ||
| Completion: 12% | ||
| Duration: 6s | ||
| Operator Status: 33 Total, 33 Available, 0 Progressing, 0 Degraded | ||
|
|
||
| Control Plane Nodes | ||
| NAME ASSESSMENT PHASE VERSION EST MESSAGE | ||
| ip-10-0-30-217.us-east-2.compute.internal Outdated Pending 4.14.0 ? | ||
| ip-10-0-53-40.us-east-2.compute.internal Outdated Pending 4.14.0 ? | ||
| ip-10-0-92-180.us-east-2.compute.internal Outdated Pending 4.14.0 ? | ||
|
|
||
| = Worker Upgrade = | ||
|
|
||
| = Worker Pool = | ||
| Worker Pool: worker | ||
| Assessment: Excluded | ||
| Completion: 0% | ||
| Worker Status: 3 Total, 3 Available, 0 Progressing, 3 Outdated, 0 Draining, 3 Excluded, 0 Degraded | ||
|
|
||
| Worker Pool Nodes | ||
| NAME ASSESSMENT PHASE VERSION EST MESSAGE | ||
| ip-10-0-20-162.us-east-2.compute.internal Excluded Paused 4.14.0 - | ||
| ip-10-0-4-159.us-east-2.compute.internal Excluded Paused 4.14.0 - | ||
| ip-10-0-99-40.us-east-2.compute.internal Excluded Paused 4.14.0 - | ||
|
|
||
| = Update Health = | ||
| Message: Outdated nodes in a paused pool 'worker' will not be updated | ||
| Since: - | ||
| Level: Warning | ||
| Impact: Update Stalled | ||
| Reference: https://docs.openshift.com/container-platform/4.15/support/troubleshooting/troubleshooting-operator-issues.html#troubleshooting-disabling-autoreboot-mco_troubleshooting-operator-issues | ||
| Resources: | ||
| MachineConfigPools: worker | ||
| Description: Pool is paused, which stops all changes to the nodes in the pool, including updates. The nodes will not be updated until the pool is unpaused by the administrator. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I would suggest to update the test name to reflect the change.
Something like:
"insights flatten linebreaks in messages" -> "insights do not flatten linebreaks in messages"