failed calling webhook "validate-exec.devworkspace-controller.svc"

[KaneTing]

Summary

I am using chectl to install eclipse-che on my ubuntu. at the last step, error occur

command:
step1:) minikube start --image-repository='registry.cn-hangzhou.aliyuncs.com/google_containers' --addons=ingress --memory=8192
step2:) chectl server:deploy --platform minikube --skip-oidc-provider-check

error:
xiaoqing@xiaoqing:~$ chectl server:deploy --platform minikube --skip-oidc-provider-check
› Installer type is set to: 'operator'
› Current Kubernetes context: 'minikube'
✔ Verify Kubernetes API...[OK]
✔ 👀 Looking for an already existing Eclipse Che instance
✔ Verify if Eclipse Che is deployed into namespace "eclipse-che"...[Not Found]
✔ ✈️ Minikube preflight checklist
✔ Verify if kubectl is installed
✔ Verify if minikube is installed
✔ Verify if minikube is running
↓ Start minikube [skipped]
→ Minikube is already running.
✔ Check Kubernetes version: [Found v1.23.3]
✔ Verify if minikube ingress addon is enabled
↓ Enable minikube ingress addon [skipped]
→ Ingress addon is already enabled.
✔ Retrieving minikube IP and domain for ingress URLs...[192.168.49.2.nip.io]
✔ Checking minikube version...[1.25.2]
↓ Check if cluster accessible [skipped]
✔ Following Eclipse Che logs
✔ Start following logs...[OK]
✔ Install Cert Manager...[Exists]
✔ Wait for Cert Manager...[OK]
✔ Create Namespace eclipse-che...[Exists]
✔ Deploy Dex
✔ Create namespace: dex...[Exists]
✔ Provide Dex certificate
✔ Read Dex certificate...[OK]
✔ Save Dex certificate...[OK: /tmp/dex-ca.crt]
✔ Add Dex certificate to Eclipse Che certificates bundle...[Exists]
✔ Create Dex service account...[Exists]
✔ Create Dex cluster role...[Exists]
✔ Create Dex cluster role binding...[Exists]
✔ Create Dex service...[Exists]
✔ Create Dex ingress...[Exists]
✔ Generate Dex username and password...[Exists]
✔ Create Dex configmap...[Exists]
✔ Create Dex deployment...[Exists]
✔ Wait for Dex is ready...[OK]
✔ Configure API server
✔ Create /etc/ca-certificates directory...[OK]
✔ Copy Dex certificate into Minikube...[OK]
✔ Configure Minikube API server...[OK]
✔ Wait for Minikube API server...[OK]
❯ 🏃‍ Running the Eclipse Che operator
✔ Create ServiceAccount che-operator in namespace eclipse-che...[Skipped: already exists]
✔ Role and RoleBindings...[OK]
✔ Create Role che-operator...[Skipped: already exists]
✔ Create RoleBinding che-operator...[Skipped: already exists]
✔ Create ClusterRole che-operator...[Skipped: already exists]
✔ Create ClusterRoleBinding che-operator...[Skipped: already exists]
✔ Create CRD checlusters.org.eclipse.che...[Skipped: already exists]
✔ Waiting 5 seconds for the new Kubernetes resources to get flushed...[OK]
✔ Create ConfigMap manager-config...[Skipped: Not found]
✔ Create Webhook Service manager-config...[Skipped: Not found]
✔ Create Certificate serving-cert...[Skipped: Not found]
✔ Create Issuer selfsigned-issuer...[Skipped: Not found]
✔ Create deployment che-operator in namespace eclipse-che...[Skipped: already exists]
❯ Operator pod bootstrap
✖ Scheduling
→ Eclipse Che operator failed, reason: InstallOrUpdateFailed, message: Internal error occurred: failed calling webhook "validate-exec.devworkspace-controller.svc": failed to call webhook: Post "https://devworkspace-webhookserver.devworkspace-controller.svc:443/validate?timeout=10s": dial tcp 10.110.45.12
…
Downloading images
Starting
Create CheCluster Custom Resource
Error: Command server:deploy failed. Error log: /home/xiaoqing/.cache/chectl/error.log.

Error log：/home/xiaoqing/.cache/chectl/error.log.
2022-04-18T12:03:10.709Z Error: Command server:deploy failed. Error log: /home/xiaoqing/.cache/chectl/error.log.
2022-04-18T12:03:10.709Z at newError (/usr/local/lib/chectl/lib/util.js:199:19)
2022-04-18T12:03:10.709Z at Object.wrapCommandError (/usr/local/lib/chectl/lib/util.js:195:12)
2022-04-18T12:03:10.709Z at Deploy. (/usr/local/lib/chectl/lib/commands/server/deploy.js:177:35)
2022-04-18T12:03:10.709Z at Generator.throw ()
2022-04-18T12:03:10.709Z at rejected (/usr/local/lib/chectl/node_modules/tslib/tslib.js:115:69)
2022-04-18T12:03:10.709Z at runMicrotasks ()
2022-04-18T12:03:10.709Z Cause: Error: Eclipse Che operator failed, reason: InstallOrUpdateFailed, message: Internal error occurred: failed calling webhook "validate-exec.devworkspace-controller.svc": failed to call webhook: Post "https://devworkspace-webhookserver.devworkspace-controller.svc:443/validate?timeout=10s": dial tcp 10.110.45.121:443: connect: connection refused. Consider increasing error recheck timeout with --k8spoderrorrechecktimeout flag.
2022-04-18T12:03:10.709Z at KubeTasks. (/usr/local/lib/chectl/lib/tasks/kube.js:48:39)
2022-04-18T12:03:10.709Z at Generator.next ()
2022-04-18T12:03:10.709Z at fulfilled (/usr/local/lib/chectl/node_modules/tslib/tslib.js:114:62)
2022-04-18T12:03:10.709Z at runMicrotasks ()

So it is confused.

My Environment:
xiaoqing@xiaoqing:~$ chectl version
chectl/7.46.0 linux-x64 node-v16.13.2

xiaoqing@xiaoqing:~$ kubelet --version
Kubernetes v1.23.5

xiaoqing@xiaoqing:~$ minikube version
minikube version: v1.25.2
commit: 362d5fdc0a3dbee389b3d3f1034e8023e72bd3a7

xiaoqing@xiaoqing:~$ minikube status
minikube
type: Control Plane
host: Running
kubelet: Running
apiserver: Running
kubeconfig: Configured

xiaoqing@xiaoqing:~$ kubectl get pod --all-namespaces

status of two of them is ImagePullBackOff
kubectl describe pod/devworkspace-controller-manager-6f75dd989-6gnsv -n devworkspace-controller

kubectl describe pod/devworkspace-webhook-server-7f6d6bffcc-klwct -n devworkspace-controller

it seems "Failed to pull image "gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0": rpc error: code = Unknown desc = Error response from daemon"

How could I do and why the error occur? thanks!

Relevant information

No response

[KaneTing]

From this test https://viewdns.info/chinesefirewall/?domain=gcr.io it should be available in China.

So Confused

[KaneTing]

I see the issue kubernetes/ingress-nginx#6335 and try pull from docker hub use:
docker pull rancher/kube-rbac-proxy:v0.5.0
docker tag rancher/kube-rbac-proxy:v0.5.0 gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0
docker rmi rancher/kube-rbac-proxy:v0.5.0
but the error still occur, so maybe it is not the problem

[KaneTing]

I see the issue #21233. it seems don't help.

[KaneTing]

I see the issue #21041 and try:

chectl server:delete --delete-namespace to clean every thing up and reinstall Eclipse Che after that. same error occur.
it seems "Failed to pull image "gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0": rpc error: code = Unknown desc = Error response from daemon"

it seems that the local image don't help.

[ibuziuk]

@KaneTing looks like the image is not accessible. Could you pull from a different registry?

[KaneTing]

@KaneTing looks like the image is not accessible. Could you pull from a different registry?

@ibuziuk how to change a different registry? the pull action is doing chectl server:deploy --platform minikube --skip-oidc-provider-check. it auto pull

[KaneTing]

@KaneTing looks like the image is not accessible. Could you pull from a different registry?

@ibuziuk actual I have pull the image to local but it can't find local and still to pull.

is it a bug?

[KaneTing]

I solve the problem of pulling image gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0.
Using docker proxy will work: https://docs.docker.com/config/daemon/systemd/#httphttps-proxy.

this is my config.
127.0.0.1:7890 is my local proxy.after change, restart docker by root.It's very import.
root> systemctl daemon-reload
root> sudo systemctl restart docker

[KaneTing]

though I can use command to pull image: docker pull gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0
but at last step: chectl server:deploy --platform minikube --skip-oidc-provider-check still occur the error.

it is so frustrating......

[KaneTing]

Although I feel very disappointed, I continue to look for solutions, because I feel that victory is at hand.
I googled: minikube failed to pull mirror, then I got some other practical inspiration：

1）https://stackoverflow.com/questions/54803393/run-minikube-start-cannot-pull-k8s-gcr-images-but-docker-pull-can-do
https://blog.csdn.net/qq_24210767/article/details/104543054
use proxy for minikube like tihs:
minikube start --registry-mirror=https://registry.docker-cn.com --insecure-registry="10.255.73.195:5000" --docker-env HTTP_PROXY=http://10.0.2.2:35033 --docker-env HTTPS_PROXY=http://10.0.2.2:35033

2）https://zhuanlan.zhihu.com/p/217324687
use command: eval $(minikube docker-env)

3. Ingress插件下载镜像失败 kubernetes/minikube#9669
   kubectl edit deployment

I chose method two to try and it worked.
docker save gcr.io/kubebuilder/kube-rbac-proxy:v0.5.0 -o gcr.io_kubebuilder_kube-rbac-proxy_v0.5.0.tag
eval $(minikube docker-env)
##https://minikube.sigs.k8s.io/docs/handbook/pushing/
docker load -i gcr.io_kubebuilder_kube-rbac-proxy_v0.5.0.tag

before eval command

after eval command

xiaoqing@xiaoqing:~/Downloads$ chectl server:deploy --platform minikube --skip-oidc-provider-check --k8spoderrorrechecktimeout=600000
(node:1805310) [DEP0123] DeprecationWarning: Setting the TLS ServerName to an IP address is not permitted by RFC 6066. This will be ignored in a future version.
(Use node --trace-deprecation ... to show where the warning was created)
› Installer type is set to: 'operator'
› Current Kubernetes context: 'minikube'
✔ Verify Kubernetes API
✔ 👀 Looking for an already existing Eclipse Che instance
✔ Verify if Eclipse Che is deployed into namespace "eclipse-che"...[Not Found]
✔ ✈️ Minikube preflight checklist
✔ Verify if kubectl is installed
✔ Verify if minikube is installed
✔ Verify if minikube is running
↓ Start minikube [skipped]
→ Minikube is already running.
✔ Check Kubernetes version: [Found v1.23.3]
✔ Verify if minikube ingress addon is enabled
↓ Enable minikube ingress addon [skipped]
→ Ingress addon is already enabled.
✔ Retrieving minikube IP and domain for ingress URLs...[192.168.49.2.nip.io]
✔ Checking minikube version...[1.25.2]
↓ Check if cluster accessible [skipped]
✔ Following Eclipse Che logs
✔ Start following logs...[OK]
✔ Install Cert Manager...[Exists]
✔ Wait for Cert Manager...[OK]
✔ Create Namespace eclipse-che...[OK]
✔ Deploy Dex
✔ Create namespace: dex...[Exists]
✔ Provide Dex certificate
✔ Read Dex certificate...[OK]
✔ Save Dex certificate...[OK: /tmp/dex-ca.crt]
✔ Add Dex certificate to Eclipse Che certificates bundle...[OK]
✔ Create Dex service account...[Exists]
✔ Create Dex cluster role...[Exists]
✔ Create Dex cluster role binding...[Exists]
✔ Create Dex service...[Exists]
✔ Create Dex ingress...[Exists]
✔ Generate Dex username and password...[Exists]
✔ Create Dex configmap...[Exists]
✔ Create Dex deployment...[Exists]
✔ Wait for Dex is ready...[OK]
✔ Configure API server
✔ Create /etc/ca-certificates directory...[OK]
✔ Copy Dex certificate into Minikube...[OK]
✔ Configure Minikube API server...[OK]
✔ Wait for Minikube API server...[OK]
✔ 🏃‍ Running the Eclipse Che operator
✔ Create ServiceAccount che-operator in namespace eclipse-che...[OK: created]
✔ Role and RoleBindings...[OK]
✔ Create Role che-operator...[OK: created]
✔ Create RoleBinding che-operator...[OK: created]
✔ Create ClusterRole che-operator...[OK: created]
✔ Create ClusterRoleBinding che-operator...[OK: created]
✔ Create CRD checlusters.org.eclipse.che...[OK: created]
✔ Waiting 5 seconds for the new Kubernetes resources to get flushed...[OK]
✔ Create ConfigMap manager-config...[Skipped: Not found]
✔ Create Webhook Service manager-config...[Skipped: Not found]
✔ Create Certificate serving-cert...[Skipped: Not found]
✔ Create Issuer selfsigned-issuer...[Skipped: Not found]
✔ Create deployment che-operator in namespace eclipse-che...[OK: created]
✔ Operator pod bootstrap
✔ Scheduling...[OK]
✔ Downloading images...[OK]
✔ Starting...[OK]
✔ Create CheCluster Custom Resource...[Created].
✔ ✅ Post installation checklist
✔ PostgreSQL pod bootstrap
✔ Scheduling...[OK]
✔ Downloading images...[OK]
✔ Starting...[OK]
✔ Devfile Registry pod bootstrap
✔ Scheduling...[OK]
✔ Downloading images...[OK]
✔ Starting...[OK]
✔ Plug-in Registry pod bootstrap
✔ Scheduling...[OK]
✔ Downloading images...[OK]
✔ Starting...[OK]
✔ Eclipse Che Dashboard pod bootstrap
✔ Scheduling...[OK]
✔ Downloading images...[OK]
✔ Starting...[OK]
✔ Eclipse Che Server pod bootstrap
✔ Scheduling...[OK]
✔ Downloading images...[OK]
✔ Starting...[OK]
✔ Eclipse Che status check...[OK]
✔ Retrieving Che self-signed CA certificate...[OK]
✔ Prepare post installation output...[OK]
✔ Show important messages
✔ Eclipse Che '7.46.0' has been successfully deployed.
✔ Documentation : https://www.eclipse.org/che/docs/
✔ -------------------------------------------------------------------------------
✔ Users Dashboard : https://192.168.49.2.nip.io/dashboard/
✔ -------------------------------------------------------------------------------
✔ Plug-in Registry : https://192.168.49.2.nip.io/plugin-registry/v3/
✔ Devfile Registry : https://192.168.49.2.nip.io/devfile-registry/
✔ -------------------------------------------------------------------------------
✔ Dex user credentials : [email protected]:admin
✔ Dex user credentials : user1@che:password
✔ Dex user credentials : user2@che:password
✔ Dex user credentials : user3@che:password
✔ Dex user credentials : user4@che:password
✔ Dex user credentials : user5@che:password
✔ -------------------------------------------------------------------------------
Command server:deploy has completed successfully in 15:28.