Can not access pods

[zzvara]

Using Windows 10 Home edition with OpenVPN GUI v10.11.0.0 & OpenVPN 2.4.6

The Kubernetes cluster is 4-node, all node with public IPs. The cluster is deployed with Kubespray, using calico. I have deployed the network-checker as well.

Server logs:

Sat Mar 30 20:09:58 2019 Running 'openvpn --config /etc/openvpn/openvpn.conf --push route 10.233.0.0 255.255.192.0 --push route 10.233.64.0 255.255.192.0 --client-config-dir /etc/openvpn/ccd --crl-verify /etc/openvpn/crl/crl.pem --status /etc/openvpn/status/server.status --status-version 2 '
Sat Mar 30 20:09:58 2019 OpenVPN 2.4.4 x86_64-alpine-linux-musl [SSL (OpenSSL)] [LZO] [LZ4] [EPOLL] [MH/PKTINFO] [AEAD] built on Nov  9 2017
Sat Mar 30 20:09:58 2019 library versions: LibreSSL 2.6.3, LZO 2.10
Sat Mar 30 20:09:58 2019 Diffie-Hellman initialized with 2048 bit key
Sat Mar 30 20:09:58 2019 Outgoing Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Sat Mar 30 20:09:58 2019 Incoming Control Channel Authentication: Using 160 bit message hash 'SHA1' for HMAC authentication
Sat Mar 30 20:09:58 2019 TUN/TAP device tun0 opened
Sat Mar 30 20:09:58 2019 TUN/TAP TX queue length set to 100
Sat Mar 30 20:09:58 2019 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Sat Mar 30 20:09:58 2019 /sbin/ip link set dev tun0 up mtu 1500
Sat Mar 30 20:09:58 2019 /sbin/ip addr add dev tun0 10.140.0.1/24 broadcast 10.140.0.255
iptables: Chain already exists.
Sat Mar 30 20:09:58 2019 Could not determine IPv4/IPv6 protocol. Using AF_INET
Sat Mar 30 20:09:58 2019 Socket Buffers: R=[87380->87380] S=[16384->16384]
Sat Mar 30 20:09:58 2019 Listening for incoming TCP connection on [AF_INET][undef]:1194
Sat Mar 30 20:09:58 2019 TCPv4_SERVER link local (bound): [AF_INET][undef]:1194
Sat Mar 30 20:09:58 2019 TCPv4_SERVER link remote: [AF_UNSPEC]
Sat Mar 30 20:09:58 2019 GID set to nogroup
Sat Mar 30 20:09:58 2019 UID set to nobody
Sat Mar 30 20:09:58 2019 MULTI: multi_init called, r=256 v=256
Sat Mar 30 20:09:58 2019 IFCONFIG POOL: base=10.140.0.2 size=252, ipv6=0
Sat Mar 30 20:09:58 2019 MULTI: TCP INIT maxclients=1024 maxevents=1028
Sat Mar 30 20:09:58 2019 Initialization Sequence Completed
Sat Mar 30 20:10:00 2019 TCP connection established with [AF_INET]10.1.38.50:54888
Sat Mar 30 20:10:01 2019 10.1.38.50:54888 TLS: Initial packet from [AF_INET]10.1.38.50:54888, sid=930e6395 3534e623
Sat Mar 30 20:10:02 2019 10.1.38.50:54888 VERIFY OK: depth=1, CN=***
Sat Mar 30 20:10:02 2019 10.1.38.50:54888 VERIFY OK: depth=0, CN=zoltan.zvara
Sat Mar 30 20:10:02 2019 10.1.38.50:54888 peer info: IV_VER=2.4.6
Sat Mar 30 20:10:02 2019 10.1.38.50:54888 peer info: IV_PLAT=win
Sat Mar 30 20:10:02 2019 10.1.38.50:54888 peer info: IV_PROTO=2
Sat Mar 30 20:10:02 2019 10.1.38.50:54888 peer info: IV_NCP=2
Sat Mar 30 20:10:02 2019 10.1.38.50:54888 peer info: IV_LZ4=1
Sat Mar 30 20:10:02 2019 10.1.38.50:54888 peer info: IV_LZ4v2=1
Sat Mar 30 20:10:02 2019 10.1.38.50:54888 peer info: IV_LZO=1
Sat Mar 30 20:10:02 2019 10.1.38.50:54888 peer info: IV_COMP_STUB=1
Sat Mar 30 20:10:02 2019 10.1.38.50:54888 peer info: IV_COMP_STUBv2=1
Sat Mar 30 20:10:02 2019 10.1.38.50:54888 peer info: IV_TCPNL=1
Sat Mar 30 20:10:02 2019 10.1.38.50:54888 peer info: IV_GUI_VER=OpenVPN_GUI_11
Sat Mar 30 20:10:02 2019 10.1.38.50:54888 WARNING: 'link-mtu' is used inconsistently, local='link-mtu 1559', remote='link-mtu 1551'
Sat Mar 30 20:10:02 2019 10.1.38.50:54888 WARNING: 'cipher' is used inconsistently, local='cipher AES-256-CBC', remote='cipher AES-256-GCM'
Sat Mar 30 20:10:02 2019 10.1.38.50:54888 WARNING: 'auth' is used inconsistently, local='auth SHA1', remote='auth [null-digest]'
Sat Mar 30 20:10:02 2019 10.1.38.50:54888 Control Channel: TLSv1.2, cipher TLSv1/SSLv3 ECDHE-RSA-AES128-GCM-SHA256, 2048 bit RSA
Sat Mar 30 20:10:02 2019 10.1.38.50:54888 [zoltan.zvara] Peer Connection Initiated with [AF_INET]10.1.38.50:54888
Sat Mar 30 20:10:02 2019 zoltan.zvara/10.1.38.50:54888 MULTI_sva: pool returned IPv4=10.140.0.2, IPv6=(Not enabled)
Sat Mar 30 20:10:02 2019 zoltan.zvara/10.1.38.50:54888 MULTI: Learn: 10.140.0.2 -> zoltan.zvara/10.1.38.50:54888
Sat Mar 30 20:10:02 2019 zoltan.zvara/10.1.38.50:54888 MULTI: primary virtual IP for zoltan.zvara/10.1.38.50:54888: 10.140.0.2
Sat Mar 30 20:10:03 2019 zoltan.zvara/10.1.38.50:54888 PUSH: Received control message: 'PUSH_REQUEST'
Sat Mar 30 20:10:03 2019 zoltan.zvara/10.1.38.50:54888 SENT CONTROL [zoltan.zvara]: 'PUSH_REPLY,block-outside-dns,dhcp-option DOMAIN svc.omega,dhcp-option DNS 10.233.0.3,route 10.233.0.0 255.255.192.0,route 10.233.64.0 255.255.192.0,route-gateway 10.140.0.1,topology subnet,ping 10,ping-restart 60,ifconfig 10.140.0.2 255.255.255.0,peer-id 0,cipher AES-256-GCM' (status=1)
Sat Mar 30 20:10:03 2019 zoltan.zvara/10.1.38.50:54888 Data Channel: using negotiated cipher 'AES-256-GCM'
Sat Mar 30 20:10:03 2019 zoltan.zvara/10.1.38.50:54888 Outgoing Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key
Sat Mar 30 20:10:03 2019 zoltan.zvara/10.1.38.50:54888 Incoming Data Channel: Cipher 'AES-256-GCM' initialized with 256 bit key

Config maps:

domain:
svc.omega
podcidr:
10.233.64.0/18
serverurl:
tcp://***.***.***.***:31214
servicecidr:
10.233.0.0/18
statusfile:
/etc/openvpn/status/server.status

• The server pushes configuration but the client would not apply them, there is nothing in the logs. I manually added push configurations from server's /etc/openvpn/openvpn.conf

These are the client configurations - excluding the keys.

client
nobind
dev tun
key-direction 1
remote-cert-tls server

script-security 2

dhcp-option DOMAIN svc.omega
dhcp-option DNS 10.233.0.3
route 10.233.0.0 255.255.192.0
route 10.233.64.0 255.255.192.0

remote ***.***.***.*** 31214 tcp

Sat Mar 30 21:07:56 2019 OpenVPN 2.4.6 x86_64-w64-mingw32 [SSL (OpenSSL)] [LZO] [LZ4] [PKCS11] [AEAD] built on Apr 26 2018
Sat Mar 30 21:07:56 2019 Windows version 6.2 (Windows 8 or greater) 64bit
Sat Mar 30 21:07:56 2019 library versions: OpenSSL 1.1.0h  27 Mar 2018, LZO 2.10
Enter Management Password:
Sat Mar 30 21:07:57 2019 TCP/UDP: Preserving recently used remote address: [AF_INET]***.***.***.***:31214
Sat Mar 30 21:07:57 2019 Attempting to establish TCP connection with [AF_INET]***.***.***.***:31214 [nonblock]
Sat Mar 30 21:07:58 2019 TCP connection established with [AF_INET]***.***.***.***:31214
Sat Mar 30 21:07:58 2019 TCP_CLIENT link local: (not bound)
Sat Mar 30 21:07:58 2019 TCP_CLIENT link remote: [AF_INET]***.***.***.***:31214
Sat Mar 30 21:07:58 2019 [***.***.***.***] Peer Connection Initiated with [AF_INET]***.***.***.***:31214
Sat Mar 30 21:07:59 2019 open_tun
Sat Mar 30 21:07:59 2019 TAP-WIN32 device [Ethernet 3] opened: \\.\Global\{99E7DDA4-169A-4248-AFB0-2C2F886D3578}.tap
Sat Mar 30 21:07:59 2019 Set TAP-Windows TUN subnet mode network/local/netmask = 10.140.0.0/10.140.0.2/255.255.255.0 [SUCCEEDED]
Sat Mar 30 21:07:59 2019 Notified TAP-Windows driver to set a DHCP IP/netmask of 10.140.0.2/255.255.255.0 on interface {99E7DDA4-169A-4248-AFB0-2C2F886D3578} [DHCP-serv: 10.140.0.254, lease-time: 31536000]
Sat Mar 30 21:07:59 2019 Successful ARP Flush on interface [15] {99E7DDA4-169A-4248-AFB0-2C2F886D3578}
Sat Mar 30 21:07:59 2019 do_ifconfig, tt->did_ifconfig_ipv6_setup=0
Sat Mar 30 21:07:59 2019 Blocking outside dns using service succeeded.
Sat Mar 30 21:08:04 2019 ROUTE: route addition failed using service: The object already exists.   [status=5010 if_index=15]
Sat Mar 30 21:08:04 2019 ROUTE: route addition failed using service: The object already exists.   [status=5010 if_index=15]
Sat Mar 30 21:08:04 2019 WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Sat Mar 30 21:08:04 2019 Initialization Sequence Completed

• Can ping and traceroute the 10.140.0.1 & 10.233.70.20 (server Pod address), but not a valid Pod on 10.233.70.**19**.

I'm using the kylemanna/openvpn on a Docker Swarm deployment, where I attach the server container to each overlay network and add the following rules to enable NAT routing:

iptables -t nat -A POSTROUTING -o eth0 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth1 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth2 -j MASQUERADE
iptables -t nat -A POSTROUTING -o eth3 -j MASQUERADE

Tried to do the same here, but did not work.

Please advice on how to proceed/debug.