Failed to initialize key-value store of network plugin: error Acquiring store lock: timed out locking store

[tamilmani1989]

What happened:

Issue:
Pod creation in AKS cluster fails with CNI error. This error happens if CNI acquires the lock and stuck in some call and not releasing the lock to another cni process. Upon investigation, found iptable list cmd in CNI taking longer time (around 2 to 3 mins) to respond. Iptables list cmd tries to do reverse dns lookup for IPAddresses present in the chain. Customer configured to use custom dns server in AKS and this custom dns server taking long to respond and thus causing CNI executing IPTable cmd to return after 2 mins and other CNI processes waiting for lock couldn't acquire it and resulting in this error.

Impact:
This affects customer who uses Azure CNI/Azure CNI Powered by Cilium pod subnet cluster and configured custom dns server and only if custom dns server is either not reponding or taking longer to respond.

Fix
CNI can prevent iptables list cmd to do reverse dns lookup and just display in numeric by running with
-n option. PR for the fix: #2682

Impacted CNI Versions: 1.4.x, 1.5.x

Orchestrator and Version (e.g. Kubernetes, Docker):
Kubernetes

Operating System (Linux/Windows):
Linux

Kernel (e.g. uanme -a for Linux or $(Get-ItemProperty -Path "C:\windows\system32\hal.dll").VersionInfo.FileVersion for Windows):
5.15

Anything else we need to know?:
[Miscellaneous information that will assist in solving the issue.]

[github-actions]

This issue is stale because it has been open for 2 weeks with no activity. Remove stale label or comment or this will be closed in 7 days

[github-actions]

Issue closed due to inactivity.