How to Create HA setup for Centos using kubeadm

[kumarganesh2814]

Is this a request for help?

yes
If yes, you should use our troubleshooting guide and community support channels, see http://kubernetes.io/docs/troubleshooting/.

If no, delete this section and continue on.

What keywords did you search in kubeadm issues before filing this one?

setup HA cluster using kubeadm

Is this a BUG REPORT or FEATURE REQUEST?

Choose one: BUG REPORT or FEATURE REQUEST
FEATURE REQUEST

Versions

kubeadm version (use kubeadm version):
kubeadm version: &version.Info{Major:"1", Minor:"8", GitVersion:"v1.8.3", GitCommit:"f0efb3cb883751c5ffdbe6d515f3cb4fbe7b7acd", GitTreeState:"clean", BuildDate:"2017-11-08T18:27:48Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"7", GitVersion:"v1.7.0", GitCommit:"d3ada0119e776222f11ec7945e6d860061339aad", GitTreeState:"clean", BuildDate:"2017-06-29T23:15:59Z", GoVersion:"go1.8.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:
  Baremetal

• OS (e.g. from /etc/os-release):
  CentOS Linux release 7.4.1708 (Core)

• Kernel (e.g. uname -a):
  3.10.0-693.5.2.el7.x86_64

• Others:
  Docker 17.11.0-ce-rc4

What happened?

Trying for Setting HA setup for kubernetes but failed everytime with diffrent issue

So if can you help me how I can create HA master setup

What you expected to happen?

How to reproduce it (as minimally and precisely as possible)?

Anything else we need to know?

[kumarganesh2814]

kubernetes/website#6295

Submitted another issue which is closed tried slack to get answer but nothing helpful yet

[luxas]

I hope you'll find the right information from this issue: #546

[kumarganesh2814]

@luxas

Hi

Today I followed instruction and created setup but somehow nodes are coming up fine

kubectl get nodes

NAME STATUS AGE VERSION
kuber-poc-app1 Ready 2h v1.8.3
kuber-poc-app2 NotReady 40m v1.8.3
kuber-poc-app3 NotReady 57m v1.8.3

Nov 20 05:21:17 kuber-poc-app2 systemd[1]: Created slice libcontainer_60557_systemd_test_default.slice.
Nov 20 05:21:17 kuber-poc-app2 systemd[1]: Starting libcontainer_60557_systemd_test_default.slice.
Nov 20 05:21:17 kuber-poc-app2 systemd[1]: Removed slice libcontainer_60557_systemd_test_default.slice.
Nov 20 05:21:17 kuber-poc-app2 systemd[1]: Stopping libcontainer_60557_systemd_test_default.slice.
Nov 20 05:21:19 kuber-poc-app2 dockerd: time="2017-11-20T05:21:19.602142669-08:00" level=info msg="Node join event for bf970f22ee17/10.127.xxxx"

systemctl status kubelet -l

● kubelet.service - kubelet: The Kubernetes Node Agent
Loaded: loaded (/etc/systemd/system/kubelet.service; enabled; vendor preset: disabled)
Drop-In: /etc/systemd/system/kubelet.service.d
└─10-kubeadm.conf
Active: active (running) since Mon 2017-11-20 05:18:06 PST; 6min ago
Docs: http://kubernetes.io/docs/
Main PID: 59242 (kubelet)
Memory: 13.1M
CGroup: /system.slice/kubelet.service
└─59242 /usr/bin/kubelet --bootstrap-kubeconfig=/etc/kubernetes/bootstrap-kubelet.conf --kubeconfig=/etc/kubernetes/kubelet.conf --pod-manifest-path=/etc/kubernetes/manifests --allow-privileged=true --network-plugin=cni --cni-conf-dir=/etc/cni/net.d --cni-bin-dir=/opt/cni/bin --cluster-dns=10.96.0.10 --cluster-domain=cluster.local --authorization-mode=Webhook --client-ca-file=/etc/kubernetes/pki/ca.crt --cadvisor-port=0 --cgroup-driver=systemd --rotate-certificates=true --cert-dir=/var/lib/kubelet/pki

Nov 20 05:18:06 kuber-poc-app2 kubelet[59242]: I1120 05:18:06.872836 59242 feature_gate.go:156] feature gates: map[]
Nov 20 05:18:06 kuber-poc-app2 kubelet[59242]: I1120 05:18:06.872951 59242 controller.go:114] kubelet config controller: starting controller
Nov 20 05:18:06 kuber-poc-app2 kubelet[59242]: I1120 05:18:06.872962 59242 controller.go:118] kubelet config controller: validating combination of defaults and flags
Nov 20 05:18:06 kuber-poc-app2 kubelet[59242]: I1120 05:18:06.879293 59242 client.go:75] Connecting to docker on unix:///var/run/docker.sock
Nov 20 05:18:06 kuber-poc-app2 kubelet[59242]: I1120 05:18:06.879378 59242 client.go:95] Start docker client with request timeout=2m0s
Nov 20 05:18:06 kuber-poc-app2 kubelet[59242]: I1120 05:18:06.891489 59242 feature_gate.go:156] feature gates: map[]
Nov 20 05:18:06 kuber-poc-app2 kubelet[59242]: W1120 05:18:06.891704 59242 server.go:289] --cloud-provider=auto-detect is deprecated. The desired cloud provider should be set explicitly
Nov 20 05:18:06 kuber-poc-app2 kubelet[59242]: I1120 05:18:06.925487 59242 certificate_manager.go:361] Requesting new certificate.

I dont see any issue too but not sure what is reson for failure

Logs from
kube-proxy
in kube-proxy-lkw7p format_color_text text_fields timer_off
https://10.127.xxx:10250/containerLogs/kube-system/kube-proxy-lkw7p/kube-proxy?timestamps=true: dial tcp 10.127.xxx:10250: getsockopt: connection refused

[kumarganesh2814]

@luxas

Cluster came up after several bounces now when I am running scale up for kube-dns,heapster and dashboard I see errors

Nov 20 09:32:59 kuber-poc-app2 kubelet[59242]: E1120 09:32:59.309467 59242 pod_workers.go:182] Error syncing pod 2d6a23ad-ce16-11e7-9b42-00505601302d ("monitoring-influxdb-67f8d587dd-9pgkd_kube-system(2d6a23ad-ce16-11e7-9b42-00505601302d)"), skipping: failed to "CreatePodSandbox" for "monitoring-influxdb-67f8d587dd-9pgkd_kube-system(2d6a23ad-ce16-11e7-9b42-00505601302d)" with CreatePodSandboxError: "CreatePodSandbox for pod "monitoring-influxdb-67f8d587dd-9pgkd_kube-system(2d6a23ad-ce16-11e7-9b42-00505601302d)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "monitoring-influxdb-67f8d587dd-9pgkd_kube-system" network: "cni0" already has an IP address different from 10.244.2.1/24"
Nov 20 09:32:59 kuber-poc-app2 kubelet[59242]: E1120 09:32:59.332467 59242 helpers.go:468] PercpuUsage had 0 cpus, but the actual number is 2; ignoring extra CPUs
Nov 20 09:32:59 kuber-poc-app2 kubelet[59242]: E1120 09:32:59.375604 59242 remote_runtime.go:92] RunPodSandbox from runtime service failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-545bc4bfd4-qw6rl_kube-system" network: "cni0" already has an IP address different from 10.244.2.1/24
Nov 20 09:32:59 kuber-poc-app2 kubelet[59242]: E1120 09:32:59.375667 59242 kuberuntime_sandbox.go:54] CreatePodSandbox for pod "kube-dns-545bc4bfd4-qw6rl_kube-system(ea1f6ae3-ce15-11e7-9b42-00505601302d)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-545bc4bfd4-qw6rl_kube-system" network: "cni0" already has an IP address different from 10.244.2.1/24
Nov 20 09:32:59 kuber-poc-app2 kubelet[59242]: E1120 09:32:59.375717 59242 kuberuntime_manager.go:633] createPodSandbox for pod "kube-dns-545bc4bfd4-qw6rl_kube-system(ea1f6ae3-ce15-11e7-9b42-00505601302d)" failed: rpc error: code = Unknown desc = NetworkPlugin cni failed to set up pod "kube-dns-545bc4bfd4-qw6rl_kube-system" network: "cni0" already has an IP address different from 10.244.2.1/24

Please advise

Best REgards
GAnesh