Alert rule etcdHighNumberOfFailedGRPCRequests in Prometheus

[aldelsa]

ISSUE TYPE

Bug Report

SUMMARY

We have a k8s cluster using kube-prometheus-stack for monitoring. This prometheus have a several alert rules to check all the cluster. One of those alert is to check request on etcd of kubernetes. The problem is that this rule is persistently alerting:
message = etcd cluster "kube-etcd": 100% of requests for Watch failed on etcd instance ". But the cluster is running properly.
We have changed the image of etcd to new one and we have the same problem.

ENVIRONMENT

Kubernetes version: 1.21.1
Etcd version: gcr.io/etcd-development/etcd:v3.4.16
Kube-prometheus-stack: 15.4.4
Operating System: Ubuntu 18.04

EXPECTED RESULTS

No receive any alert about etcd

ACTUAL RESULTS
Labels
alertname = etcdHighNumberOfFailedGRPCRequests
grpc_method = Watch
grpc_service = etcdserverpb.Watch
instance = 192.168.251.221:2381
job = kube-etcd
prometheus = monitoring/kube-prometheus-stack-prometheus
severity = warning
Annotations
message = etcd cluster "kube-etcd": 100% of requests for Watch failed on etcd instance 192.168.251.221:2381.

[ksa-real]

I wonder if #9166 is related

[ksa-real]

I see that according to Prometheus data the grpc calls to etcd's Watch method have either "Cancelled" or "Unknown" grpc code. Check the graph of below expression e.g. last 24 hours:

sum by (grpc_code) (grpc_server_handled_total{job=~".*etcd.*",grpc_service="etcdserverpb.Watch",instance="10.10.10.5:2379",grpc_method="Watch", job="kube-etcd"})

So effectively if there are no calls to Watch API, the service looks healthy. If there are calls, they fail with !="OK" grpc code, and Prometheus produces an alert. Can someone comment on whether this behavior is expected?

[allenporter]

I believe when I've looked into this before it is #10289

[allenporter]

(The fix appears in the etcd 3.5 changelog, though I have not upgraded yet)

[allenporter]

@lilic updated the rules for etcdHighNumberOfFailedGRPCRequests in #13127

roughly:

grpc_code=~"Unknown|FailedPrecondition|ResourceExhausted|Internal|Unavailable|DataLoss|DeadlineExceeded"

Now i think upstream projects that copied these rules just need to be updated. Great!

[allenporter]

kube-prometheus-stack is currently not able to be easily updated with the latest etcd rules due to prometheus-community/helm-charts#225

[allenporter]

I think this issue is fixed and can be closed.

[spzala]

@allenporter thanks for findings!
@aldelsa closing the issue per @allenporter comment and #13127 Please free to reopen if needed. Thanks!