Disable etcd alert about false positive cluster state

[surajssd]

Following alert is triggered when you have multiple gRPC failures on watch request.

[firing]  — etcdHighNumberOfFailedGRPCRequests (1)
critical
etcd cluster "kube-etcd": 100% of requests for Watch failed on etcd instance 10.99.170.163:2381.
Source
Labels:
· alertname: etcdHighNumberOfFailedGRPCRequests
· grpc_method: Watch
· grpc_service: etcdserverpb.Watch
· instance: 10.1.1.1:2381
· job: kube-etcd
· prometheus: monitoring/prometheus-operator-prometheus
· severity: critical

• This is being fixed in etcd here: etcdserver: fix incorrect metrics generated when clients cancel watches etcd-io/etcd#12196.
• Relevant issue: gRPC code Unavailable instead Canceled etcd-io/etcd#10289.

[invidian]

I investigated this issue privately and it seems removing the alert should be okay'ish for the time being.

Alternatively we could modify the formula to ignore offending metrics, leaving others in place.

Then, we should restore the alerts once issue is fixed in etcd.