Repository navigation
Create runbook for troubleshooting AKS upgrade failures - #611
Conversation
Adding a new folder under "runbooks" for upgrade issues + new runbook for troubleshooting AKS upgrade failures.
WalkthroughThe changes update the runbook catalog by revising an existing runbook's description and adding a new entry for AKS upgrade troubleshooting. A detailed markdown guide for diagnosing and resolving Azure Kubernetes Service upgrade failures is introduced, outlining step-by-step procedures for common error scenarios. Changes
Sequence Diagram(s)sequenceDiagram
participant User
participant RunbookCatalog
participant MarkdownGuide
User->>RunbookCatalog: Search for AKS upgrade troubleshooting runbook
RunbookCatalog-->>User: Provide runbook entry and link to markdown guide
User->>MarkdownGuide: Open AKS Upgrade Troubleshooting Guidelines
MarkdownGuide-->>User: Display diagnostic steps and remediation advice
Possibly related PRs
Suggested reviewers
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. 🪧 TipsChatThere are 3 ways to chat with CodeRabbit:
SupportNeed help? Create a ticket on our support page for assistance with any issues or questions. Note: Be mindful of the bot's finite context window. It's strongly recommended to break down tasks such as reading entire modules into smaller chunks. For a focused discussion, use review comments to chat about specific files and their changes, instead of using the PR comments. CodeRabbit Commands (Invoked using PR comments)
Other keywords and placeholders
CodeRabbit Configuration File (
|
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 1
🔭 Outside diff range comments (1)
holmes/plugins/runbooks/upgrade/upgrade_troubleshooting_instructions (1)
48-48: Complete the incomplete line.Line 48 appears to be incomplete and cuts off mid-sentence.
The file appears to be incomplete. Please complete the final line or remove it if it was added in error.
🧹 Nitpick comments (4)
holmes/plugins/runbooks/upgrade/upgrade_troubleshooting_instructions (4)
8-8: Clarify the instruction for AI behavior.The phrase "Instead of providing next steps to the user, you need to follow the troubleshoot guide to execute the steps" is ambiguous and could be confusing. It's unclear whether the AI should execute steps directly or guide users through execution.
Consider rephrasing for clarity:
-* Instead of providing next steps to the user, you need to follow the troubleshoot guide to execute the steps. +* Follow this troubleshooting guide systematically to diagnose issues and provide specific remediation guidance to users.
18-18: Fix formatting and grammar issues.This line has formatting issues and grammatical errors that affect readability.
Apply this diff to improve clarity and formatting:
-If the cluster is in a Succeeded state, follow the steps below to identify if there are any potential issues which could results in a failed cluster - for example, check for any restrictive PDBs or that outbound connectivity endpoints are blocked. If the cluster is in Failed state, check the error message and follow the steps below for that specific error code. +If the cluster is in a Succeeded state, follow the steps below to identify potential issues that could result in cluster upgrade failures (e.g., restrictive PDBs or blocked outbound connectivity endpoints). If the cluster is in a Failed state, check the error message and follow the steps below for that specific error code.
26-26: Enhance PDB troubleshooting guidance.The current instruction is good but could be more specific about what constitutes a "restrictive PDB" beyond just MaxUnavailable=0.
Consider expanding the guidance:
-* If the upgrade fails due to one or more restrictive PDB, this means that one or more PDBs were set in the cluster which do not allow for any pod disruptions. Get all the PDBs configured on the cluster by running 'kubectl get pdb' and identify all PDBs where MaxUnavailable is set to 0 - if any are found, call them out specifically and ask the customer to follow the guidance in this troubleshooting guide for restrictive PDBs: https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/error-codes/unsatisfiablepdb-error. +* If the upgrade fails due to one or more restrictive PDB, this means that one or more PDBs were set in the cluster which do not allow for any pod disruptions. Get all the PDBs configured on the cluster by running 'kubectl get pdb -A' and identify all PDBs where MaxUnavailable is set to 0 or MinAvailable equals the total number of replicas - if any are found, call them out specifically and ask the customer to follow the guidance in this troubleshooting guide for restrictive PDBs: https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/error-codes/unsatisfiablepdb-error.
23-23: Enhance NSG analysis guidance.The current instruction is good but could benefit from more specific guidance on how to identify problematic NSG rules.
Consider adding more specific guidance:
-* Check if the outbound connectivity issue is due to an NSG rule by first getting all of the NSGs in the current subscription. Find the NSGs which have a ResourceGroup which contains both the current resource group and cluster name. For each NSG, list all of the rules and determine if any rule is blocking an essential endpoint in the doc linked above. Do this analysis on behalf of the user and only return the problematic NSG rules. +* Check if the outbound connectivity issue is due to an NSG rule by first getting all of the NSGs in the current subscription. Find the NSGs which have a ResourceGroup which contains both the current resource group and cluster name. For each NSG, list all of the rules and determine if any rule is blocking an essential endpoint in the doc linked above. Pay special attention to rules that DENY traffic to ports 443, 9000, or 1194, or block access to *.hcp.<region>.azmk8s.io, mcr.microsoft.com, or *.data.mcr.microsoft.com. Do this analysis on behalf of the user and only return the problematic NSG rules.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
holmes/plugins/runbooks/upgrade/upgrade_troubleshooting_instructions(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: llm_evals
- GitHub Check: Pre-commit checks
🔇 Additional comments (1)
holmes/plugins/runbooks/upgrade/upgrade_troubleshooting_instructions (1)
1-47: Overall documentation structure and completeness.The runbook provides a well-structured approach to AKS upgrade troubleshooting with good coverage of common failure scenarios. The workflow is logical and the references to Microsoft documentation are appropriate.
The documentation effectively addresses the key AKS upgrade failure scenarios and provides actionable guidance for troubleshooting.
Adding new AKS upgrade failures runbook to catalog.json with latest update being 7/8/2025.
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (4)
holmes/plugins/runbooks/upgrade/upgrade_troubleshooting_instructions (4)
18-18: Fix grammatical error in the conditional statement.There's a grammatical error that affects readability.
-If the cluster is in a Succeeded state, follow the steps below to identify if there are any potential issues which could results in a failed cluster - for example, check for any restrictive PDBs or that outbound connectivity endpoints are blocked. +If the cluster is in a Succeeded state, follow the steps below to identify if there are any potential issues which could result in a failed cluster - for example, check for any restrictive PDBs or that outbound connectivity endpoints are blocked.
22-22: Provide more specific guidance on identifying blocking NSG rules.The instruction to "determine if any rule is blocking an essential endpoint" could be more specific about what constitutes a blocking rule to help users identify the actual problem.
Consider adding more specific criteria for identifying blocking rules:
-For each NSG, list all of the rules and determine if any rule is blocking an essential endpoint in the doc linked above. +For each NSG, list all of the rules and determine if any rule is blocking an essential endpoint in the doc linked above. Look for rules with 'Deny' actions that target the required outbound ports (443, 80, 9000, 1194) or FQDN endpoints.
28-28: Enhance support ticket guidance with specific steps.The guidance on opening a support ticket could be more actionable by providing specific steps for the Azure Portal process.
Consider adding more detailed steps for the support ticket process:
-You will need to raise the limit or quota for your subscription by filing a "Service and subscription limits (quotas)" support ticket to increase the quota for compute cores. Provide detailed instructions in the response on how to open a specific "Service and subscription limits" support ticket through the Azure Portal for an AKS cluster. +You will need to raise the limit or quota for your subscription by filing a "Service and subscription limits (quotas)" support ticket to increase the quota for compute cores. In the Azure Portal, navigate to Help + Support → New support request → Issue type: Service and subscription limits (quotas) → Quota type: Compute-VM (cores-vCPUs) subscription limit increases, then specify the required quota increase for your AKS cluster's region and VM series.
45-45: Add missing line break at end of file.The file appears to be missing a line break at the end, which is a common formatting convention.
* If upgrade is failing due to other issues, provide specific guidance from the documentation towards reviewing the upgrade logs and configuration. +
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
holmes/plugins/runbooks/upgrade/upgrade_troubleshooting_instructions(1 hunks)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (2)
- GitHub Check: llm_evals
- GitHub Check: Pre-commit checks
🔇 Additional comments (1)
holmes/plugins/runbooks/upgrade/upgrade_troubleshooting_instructions (1)
1-45: Well-structured troubleshooting guide with comprehensive coverage.The runbook provides excellent structured guidance for AKS upgrade troubleshooting:
- Clear workflow with logical progression
- Covers major failure scenarios (connectivity, PDB, quota)
- Emphasizes using official Microsoft documentation
- Provides actionable diagnostic steps
- Includes specific error codes and remediation approaches
The guide effectively balances thoroughness with practical usability for troubleshooting AKS upgrade failures.
|
@julia-yin I think the runbook file name is missing the |
…g_instructions.md
There was a problem hiding this comment.
Actionable comments posted: 0
🧹 Nitpick comments (4)
holmes/plugins/runbooks/upgrade/upgrade_troubleshooting_instructions.md (4)
6-8: Bullet wording & consistencyMinor grammar and wording tweaks improve clarity and keep terminology consistent with the rest of the doc (“troubleshooting guide”).
-* Instead of providing next steps to the user, you need to follow the troubleshoot guide to execute the steps. +* Instead of providing next steps to the user, follow the troubleshooting guide and execute each step yourself.
18-18: Grammar: “results” → “result”, dash → em-dash-…issues which could results in a failed cluster - for example, check… +…issues that could result in a failed cluster—for example, check…This also fixes the repeated “which” construction flagged by the linter.
21-22: Replace bare URL with descriptive link & trim verbosityLong bare URLs trip MD034 and hurt readability. Embed them with descriptive text and split very long sentences:
-…Check if any of the required rules/endpoints in https://learn.microsoft.com/en-us/azure/aks/outbound-rules-control-egress#azure-global-required-network-rules are blocked due to a network policy, NSG, or Azure firewall and identify the problematic rule. +…Verify whether any of the required outbound-rules listed in +[AKS outbound connectivity requirements]( +https://learn.microsoft.com/en-us/azure/aks/outbound-rules-control-egress#azure-global-required-network-rules) +are blocked by a network policy, NSG, or Azure Firewall, and identify the offending rule.This satisfies MD034 and tightens the prose.
37-44: Fix sub-list indentation (MD007)Markdown-lint expects child list items to be indented by exactly two spaces from the parent bullet. Current four-space indent causes MD007 failures.
-* Based on the findings, suggest which sections of the documentation are most relevant. - * If upgrade is failing due to PDB blocking, provide specific guidance from the documentation towards adjusting PDB settings. - * If upgrade is failing due to quota exhaustion, provide specific guidance from the documentation towards reviewing resource quotas. - * If upgrade is failing due to node issues, provide specific guidance from the documentation towards reviewing node status and health. - * If upgrade is failing due to network issues, provide specific guidance from the documentation towards reviewing network configuration. - * If upgrade is failing due to other issues, provide specific guidance from the documentation towards reviewing the upgrade logs and configuration. +* Based on the findings, suggest which sections of the documentation are most relevant. + * If upgrade is failing due to PDB blocking, provide specific guidance from the documentation towards adjusting PDB settings. + * If upgrade is failing due to quota exhaustion, provide specific guidance from the documentation towards reviewing resource quotas. + * If upgrade is failing due to node issues, provide specific guidance from the documentation towards reviewing node status and health. + * If upgrade is failing due to network issues, provide specific guidance from the documentation towards reviewing network configuration. + * If upgrade is failing due to other issues, provide specific guidance from the documentation towards reviewing the upgrade logs and configuration.
📜 Review details
Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro
📒 Files selected for processing (1)
holmes/plugins/runbooks/upgrade/upgrade_troubleshooting_instructions.md(1 hunks)
🧰 Additional context used
🪛 LanguageTool
holmes/plugins/runbooks/upgrade/upgrade_troubleshooting_instructions.md
[grammar] ~1-~1: Use correct spacing
Context: # AKS Upgrade Troubleshooting Guidelines ## Goal Your primary goal when using these...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~4-~4: Use correct spacing
Context: ... following the workflow for AKS upgrade diagnosis. * Use the tools to gather information abo...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~8-~8: There might be a mistake here.
Context: ...eps to the user, you need to follow the troubleshoot guide to execute the steps. ## Workflo...
(QB_NEW_EN_OTHER)
[grammar] ~8-~8: Use correct spacing
Context: ...w the troubleshoot guide to execute the steps. ## Workflow for AKS Upgrade Diagnosis 1. ...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~10-~10: Use correct spacing
Context: ...the steps. ## Workflow for AKS Upgrade Diagnosis 1. Check Cluster and Nodepool Status: ...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[typographical] ~13-~13: To join two clauses or set off examples, consider using an em dash.
Context: ...atus:** * Get the current cluster context - cluster name, resource group, and subscr...
(QB_NEW_EN_DASH_RULE_EM)
[grammar] ~15-~15: There might be a problem here.
Context: ... Check the provisioning status of your AKS nodepools. If any nodepools have a Failed status, ...
(QB_NEW_EN_MERGED_MATCH)
[grammar] ~16-~16: Use correct spacing
Context: ...he error message details (starting with aka.ms). If the cluster is in a Succeeded state,...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[style] ~17-~17: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...essage details (starting with aka.ms). If the cluster is in a Succeeded state, fo...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[grammar] ~18-~18: Know when to use “that” and “which”
Context: ...ntify if there are any potential issues which could results in a failed cluster - for...
(QB_NEW_EN_OTHER_ERROR_IDS_3)
[grammar] ~18-~18: Make sure you are using the right part of speech
Context: ...re are any potential issues which could results in a failed cluster - for example, chec...
(QB_NEW_EN_OTHER_ERROR_IDS_21)
[typographical] ~18-~18: To join two clauses or set off examples, consider using an em dash.
Context: ... issues which could results in a failed cluster - for example, check for any restrictive P...
(QB_NEW_EN_DASH_RULE_EM)
[style] ~18-~18: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...und connectivity endpoints are blocked. If the cluster is in Failed state, check t...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[grammar] ~18-~18: Use articles correctly
Context: ...ndpoints are blocked. If the cluster is in Failed state, check the error message an...
(QB_NEW_EN_OTHER_ERROR_IDS_11)
[grammar] ~18-~18: Use correct spacing
Context: ...the steps below for that specific error code. 2. **Error code: 'VMExtensionError_OutBoundC...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~21-~21: There might be a mistake here.
Context: ...tains 'VMExtensionError_OutBoundConnFail', this means that the cluster upgrade fai...
(QB_NEW_EN_OTHER)
[grammar] ~21-~21: There might be a mistake here.
Context: ...tion in a network policy, NSG, or Azure firewall which is denying traffic to the endpoin...
(QB_NEW_EN_OTHER)
[grammar] ~21-~21: Add a comma
Context: ... due to a network policy, NSG, or Azure firewall and identify the problematic rule. ...
(QB_NEW_EN_OTHER_ERROR_IDS_22)
[style] ~22-~22: Consider removing “of” to be more concise
Context: ... is due to an NSG rule by first getting all of the NSGs in the current subscription. Find ...
(ALL_OF_THE)
[grammar] ~22-~22: Know when to use “that” and “which”
Context: ...the current subscription. Find the NSGs which have a ResourceGroup which contains bot...
(QB_NEW_EN_OTHER_ERROR_IDS_3)
[grammar] ~22-~22: Know when to use “that” and “which”
Context: ...ind the NSGs which have a ResourceGroup which contains both the current resource grou...
(QB_NEW_EN_OTHER_ERROR_IDS_3)
[style] ~22-~22: Consider removing “of” to be more concise
Context: ...up and cluster name. For each NSG, list all of the rules and determine if any rule is bloc...
(ALL_OF_THE)
[grammar] ~22-~22: Use correct spacing
Context: ...ser and only return the problematic NSG rules. 3. **PDB blocking upgrade: Error code "Unsat...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~25-~25: Make sure to use plural and singular nouns correctly
Context: ...de fails due to one or more restrictive PDB, this means that one or more PDBs were ...
(QB_NEW_EN_OTHER_ERROR_IDS_10)
[grammar] ~25-~25: Know when to use “that” and “which”
Context: ...ne or more PDBs were set in the cluster which do not allow for any pod disruptions. G...
(QB_NEW_EN_OTHER_ERROR_IDS_3)
[typographical] ~25-~25: To join two clauses or set off examples, consider using an em dash.
Context: ...all PDBs where MaxUnavailable is set to 0 - if any are found, call them out specific...
(QB_NEW_EN_DASH_RULE_EM)
[grammar] ~25-~25: Use correct spacing
Context: ...bleshooting guide for restrictive PDBs: https://learn.microsoft.com/en-us/troubleshoot/azure/azure-kubernetes/error-codes/unsatisfiablepdb-error. 4. **Quota exhaustion issues: Error code "Qu...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~28-~28: There might be a mistake here.
Context: ...ent quota with error code "QuotaExceeded", this means that your subscription doesn...
(QB_NEW_EN_OTHER)
[grammar] ~28-~28: Use correct spacing
Context: ...ket through the Azure Portal for an AKS cluster. ## Synthesize Findings Based on the output...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~33-~33: Use articles correctly
Context: ...emove the blocking NSG rule x and retry upgrade." * If upgrade is failing due to a rest...
(QB_NEW_EN_OTHER_ERROR_IDS_11)
[grammar] ~34-~34: Use articles correctly
Context: ...g NSG rule x and retry upgrade." * If upgrade is failing due to a restrictive PDB, ch...
(QB_NEW_EN_OTHER_ERROR_IDS_11)
[grammar] ~34-~34: There might be a mistake here.
Context: ...ling due to a restrictive PDB on your x pods which does not tolerate any disruptions...
(QB_NEW_EN_OTHER)
[grammar] ~34-~34: Use correct spacing
Context: ...ime to allow upgrades while maintaining availability." ## Recommend Remediation Steps (Based on D...
(QB_NEW_EN_OTHER_ERROR_IDS_5)
[grammar] ~40-~40: Use articles correctly
Context: ...mentation are most relevant. * If upgrade is failing due to PDB blocking, provide...
(QB_NEW_EN_OTHER_ERROR_IDS_11)
[style] ~42-~42: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...ards reviewing resource quotas. * If upgrade is failing due to node issues, ...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[grammar] ~42-~42: Use articles correctly
Context: ...s reviewing resource quotas. * If upgrade is failing due to node issues, provide ...
(QB_NEW_EN_OTHER_ERROR_IDS_11)
[style] ~43-~43: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...viewing node status and health. * If upgrade is failing due to network issue...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[grammar] ~43-~43: Use articles correctly
Context: ...wing node status and health. * If upgrade is failing due to network issues, provi...
(QB_NEW_EN_OTHER_ERROR_IDS_11)
[style] ~44-~44: Three successive sentences begin with the same word. Consider rewording the sentence or use a thesaurus to find a synonym.
Context: ...eviewing network configuration. * If upgrade is failing due to other issues,...
(ENGLISH_WORD_REPEAT_BEGINNING_RULE)
[grammar] ~44-~44: Use articles correctly
Context: ...ewing network configuration. * If upgrade is failing due to other issues, provide...
(QB_NEW_EN_OTHER_ERROR_IDS_11)
🪛 markdownlint-cli2 (0.17.2)
holmes/plugins/runbooks/upgrade/upgrade_troubleshooting_instructions.md
21-21: Bare URL used
(MD034, no-bare-urls)
25-25: Bare URL used
(MD034, no-bare-urls)
37-37: Bare URL used
(MD034, no-bare-urls)
40-40: Unordered list indentation
Expected: 2; Actual: 4
(MD007, ul-indent)
41-41: Unordered list indentation
Expected: 2; Actual: 4
(MD007, ul-indent)
42-42: Unordered list indentation
Expected: 2; Actual: 4
(MD007, ul-indent)
43-43: Unordered list indentation
Expected: 2; Actual: 4
(MD007, ul-indent)
44-44: Unordered list indentation
Expected: 2; Actual: 4
(MD007, ul-indent)
⏰ Context from checks skipped due to timeout of 90000ms. You can increase the timeout in your CodeRabbit configuration to a maximum of 15 minutes (900000ms). (3)
- GitHub Check: build (3.11)
- GitHub Check: build (3.12)
- GitHub Check: build (3.10)
|
I am going to merge this runbook though we don't have a standard format of runbook yet. |
3 participants
Adding a new folder under "runbooks" for upgrade issues + new runbook for troubleshooting AKS upgrade failures.