Reflector should backoff with Jitter to avoid global synchronization during API server OOM

[zhan849]

What happened:
We recently had a production issue where one of our large cluster (~3k nodes) has it's API server OOMKilled. This caused all kubelets' connection to master broken, and caused a global synchronization of kubelet's reconnecting to master. Due to the burst of API call (18K QPS, 7x than normal), API server went into even worse condition, and caused more replicas to get OOMKilled.

Current reflector impl has fixed 1sec backoff before calling next ListWatch, which shall be the root cause of such instability.

What you expected to happen:
reflectors to backoff with jitter

How to reproduce it (as minimally and precisely as possible):

1. create a cluster with a few thousands of nodes
2. kill api server and reboot it

Anything else we need to know?:

Environment:

• Kubernetes version (use kubectl version):
• Cloud provider or hardware configuration:
• OS (e.g: cat /etc/os-release):
• Kernel (e.g. uname -a):
• Install tools:
• Network plugin and version (if this is a network-related bug):
• Others:

/assign

/sig api-machinery scalability

[jktomer]

/cc @lavalamp