Failing to access services using NodePort on Linux

[marlon-sousa]

Hello,

I recently made a new Arch Linux installation and now can no longer access services exposed via nodeports.

The old machine still accepts requests as usual, but not this one.

The same services are accessible on the new setup via kubectl port-forward on localhost, so I know that they are working.

Trying to access any service on minikub ip (192.168.49.2) returns connection refused.

Enabling or disabling iptables, flushing all its rules also doesn't seen to make a difference.

I am opening this issue for two reasons:

1. Ask for help to try to figure out if there is a solution.
2. Let you know that may be some regression happened, although I am not sure.

Below, my command to start minikube:

minikube start --driver=docker --mount --mount-string $(PWD)/volumes:/volumes

test deployment

A simple deployment to test goes like this:

apiVersion: v1
kind: ConfigMap
metadata:
  name: postgres-config
  labels:
    app: postgres
data:
  POSTGRES_USER: postgresadmin
  POSTGRES_PASSWORD: admin123
---
apiVersion: v1
kind: PersistentVolume
metadata:
  name: postgres-pv-volume
  labels:
    type: local
    app: postgres
spec:
  storageClassName: manual
  capacity:
    storage: 100Mi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/volumes/db"
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: postgres-pv-claim
  labels:
    app: postgres
spec:
  storageClassName: manual
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 100Mi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: postgres
spec:
  replicas: 1
  selector:
    matchLabels:
      app: postgres
  template:
    metadata:
      labels:
        app: postgres
    spec:
      containers:
        - name: postgres
          image: postgres:latest
          imagePullPolicy: "IfNotPresent"
          ports:
            - containerPort: 5432
          envFrom:
            - configMapRef:
                name: postgres-config
          volumeMounts:
            - mountPath: /var/lib/postgresql/data
              name: postgredb
      volumes:
        - name: postgredb
          persistentVolumeClaim:
            claimName: postgres-pv-claim
---
apiVersion: v1
kind: Service
metadata:
  name: postgres
  labels:
    app: postgres
spec:
  type: NodePort
  ports:
    - port: 5432
      nodePort: 30001
  selector:
    app: postgres

Environment information

Docker

Client:
 Version:           20.10.6
 API version:       1.41
 Go version:        go1.16.3
 Git commit:        370c28948e
 Built:             Mon Apr 12 14:10:41 2021
 OS/Arch:           linux/amd64
 Context:           default
 Experimental:      true

Server:
 Engine:
  Version:          20.10.6
  API version:      1.41 (minimum version 1.12)
  Go version:       go1.16.3
  Git commit:       8728dd246c
  Built:            Mon Apr 12 14:10:25 2021
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          v1.5.2
  GitCommit:        36cc874494a56a253cd181a1a685b44b58a2e34a.m
 runc:
  Version:          1.0.0-rc95
  GitCommit:        b9ee9c6314599f1b4a7f497e1f1f856fe433d3b7
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0

Minikube

minikube version: v1.20.0
commit: c61663e942ec43b20

kubectl

Client Version: version.Info{Major:"1", Minor:"21", GitVersion:"v1.21.1", GitCommit:"5e58841cce77d4bc13713ad2b91fa0d961e69192", GitTreeState:"archive", BuildDate:"2021-05-14T14:09:09Z", GoVersion:"go1.16.4", Compiler:"gc", Platform:"linux/amd64"}
Server Version: version.Info{Major:"1", Minor:"20", GitVersion:"v1.20.2", GitCommit:"faecb196815e248d3ecfb03c680a4507229c2a56", GitTreeState:"clean", BuildDate:"2021-01-13T13:20:00Z", GoVersion:"go1.15.5", Compiler:"gc", Platform:"linux/amd64"}

Arch Linux

This is the iso 2020-06-01

Extra information

kubectl logs on the pod didn't show nothing special, there were no postgresql errors. Similarly, kubectl describe on the service and on the deployment didn't list errors on the events section.

I don't know how to inspect logs from minikube itself so I don't know if errors happened some where else.

minikube ssh worked as expected. However, just to try to play from the safe side, within the ssh I tryed to access service on localhost 30001 and also didn't have success.

I gave postgres as an example, but this is the same behavior of other services.

From my point of view docker isn't, somehow, delivering requests to high ports. This is similar to a behaviour I had with Mac OS, but now its happening on Linux and it shouldn't. Do you have any hint as how I can at least try to debug it further of if you know a solution?

Thanks,
Marlon

[marlon-sousa]

Interestingly I have found that pods also can not access services (dns does not resolve) in this configuration. It really seems to be some docker issue but I have no idea what it could be.

[zhan9san]

Hi @marlon-sousa
I am glad to help you figure it out.

In a nutshell, it's a known issue, #11418

How to fix it temporarily

sudo sysctl net/netfilter/nf_conntrack_max=131072
minikube delete
minikube start --driver=docker

Generally, this kind of access issue is related to kube-proxy.

Here is my issue finding prodcedure.

1. Create a pure environment.

$ cat Vagrantfile
Vagrant.configure("2") do |config|
  config.vm.box = "archlinux/archlinux"
  config.vm.network "public_network", bridge: "en0: Wi-Fi (Wireless)"
  config.vm.provider "virtualbox" do |vb|
      vb.memory = "4096"
      vb.cpus = 2
  end
end

Create an Arch Linux VM

vagrant up

2. Install and start Minikube and Docker

Login to Arch Linux VM

vagrant ssh

sudo pacman -Syy docker
sudo pacman -Syy minikube
sudo usermod -aG docker vagrant
sudo systemctl start docker
minikube start --driver=docker

3. Create a minimal service to limit the issue scope and try to reproduce it.

cat test.yml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-deployment
spec:
  selector:
    matchLabels:
      app: metrics
      department: engineering
  replicas: 1
  template:
    metadata:
      labels:
        app: metrics
        department: engineering
    spec:
      containers:
      - name: hello
        image: "gcr.io/google-samples/hello-app:2.0"
        env:
        - name: "PORT"
          value: "8080"

---
apiVersion: v1
kind: Service
metadata:
  name: my-np-service
spec:
  type: NodePort
  selector:
    app: metrics
    department: engineering
  ports:
  - protocol: TCP
    port: 8080
    nodePort: 30080

a. Access this NodePort service. It failed.

kubectl run -it --rm --image=curlimages/curl --restart=Never one-time-curl -- http://192.168.49.2:30080

b. Access service in pod. It succeeded.

$ kubectl exec -it my-deployment-6ccc959b54-hvgx4 -- sh
$ wget http://127.0.0.1:8008

It means that service works in pod level.

Try to get more information

kube-proxy pod doesn't work.

[vagrant@archlinux ~]$ kubectl get po -A
NAMESPACE     NAME                               READY   STATUS             RESTARTS   AGE
kube-system   coredns-74ff55c5b-z2f4l            0/1     Running            0          11s
kube-system   etcd-minikube                      0/1     Running            0          20s
kube-system   kube-apiserver-minikube            1/1     Running            0          20s
kube-system   kube-controller-manager-minikube   0/1     Running            0          20s
kube-system   kube-proxy-ghs4x                   0/1     CrashLoopBackOff   1          11s
kube-system   kube-scheduler-minikube            0/1     Running            0          20s
kube-system   storage-provisioner                0/1     Error              1          25s

$ kubectl logs -n kube-system kube-proxy-ghs4x
I0607 04:48:16.632051       1 node.go:172] Successfully retrieved node IP: 192.168.49.2
I0607 04:48:16.632101       1 server_others.go:142] kube-proxy node IP is an IPv4 address (192.168.49.2), assume IPv4 operation
W0607 04:48:16.666122       1 server_others.go:578] Unknown proxy mode "", assuming iptables proxy
I0607 04:48:16.666633       1 server_others.go:185] Using iptables Proxier.
I0607 04:48:16.667313       1 server.go:650] Version: v1.20.2
I0607 04:48:16.667822       1 conntrack.go:100] Set sysctl 'net/netfilter/nf_conntrack_max' to 131072
F0607 04:48:16.667986       1 server.go:495] open /proc/sys/net/netfilter/nf_conntrack_max: permission denied

Fix it

sudo sysctl net/netfilter/nf_conntrack_max=131072
minikube delete
minikube start --driver=docker

All works well.
Enjoy it.

[andriyDev]

@marlon-sousa Did @zhan9san's solution work?

[spowelljr]

@marlon-sousa Due to no response I'm going to close this issue.