Policy validation webhook timing out

[danielmilanov]

What is the issue?

The policy validation webhook times out a number of times every day.
Internal error occurred: failed calling webhook "linkerd-policy-validator.linkerd.io": failed to call webhook: Post "https://linkerd-policy-validator.ns-linkerd.svc:443/?timeout=20s": context deadline exceeded

Issue persists after timeout was doubled to 20s. There are no linkerd-destination workload restarts, CPU throttling, or any other disruptions that can be traced.

The admission request events are triggered by flux dry-run calls.

How can it be reproduced?

Unable to reproduce the issue manually with matching policy resources, and server side dry-run calls.

Logs, error output, etc

Nothing is logged at the info level after service startup

output of linkerd check -o short

linkerd-existence
-----------------
× 'linkerd-config' config map exists
    configmaps "linkerd-config" not found
    see https://linkerd.io/2.14/checks/#l5d-existence-linkerd-config for hints

Status check results are ×

Environment

K8s v1.29.8
AWS EKS
linkerd stable-2.14.10
Host OS: EKS Bottlerocket

Possible solution

No response

Additional context

No response

Would you like to work on fixing this bug?

None

[olix0r]

What types of resources are being validated? This will help us narrow down if there are specific calls that can block.