Docker 18.03.1-ce routing mesh not working

[deminngi]

TLDR

@fcrisciani I have read and understood how mesh is working using ingress network and ingress_sbox on docker_gwbridge and using a custom overlay network for services.

Read a lot of stuff and checked everything out (see docker-support.log) I cannot reach the published external port. To avoid side effects my firewallD is not running on the swarm cluster.

Issue type

Installing nginx with replica 1 cannot be accessed via the published port 80, suppose
routing mesh is not working

Expect

Can access published port for every node with w3m http://localhost

Got

Cannot load http://localhost

OS Version/build

Kernel: 4.4.x
OS: Ubuntu 18.04 LTS
Arch: arm64
Docker: 18.03.1-ce

API version

Client:
Version: 18.03.1-ce
API version: 1.37
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:16:22 2018
OS/Arch: linux/arm64
Experimental: false
Orchestrator: swarm

Server:
Engine:
Version: 18.03.1-ce
API version: 1.37 (minimum version 1.12)
Go version: go1.9.5
Git commit: 9ee9f40
Built: Thu Apr 26 07:14:27 2018
OS/Arch: linux/arm64
Experimental: true

Steps to reproduce

- Create named volume
   docker volume create --opt type=none --opt device=/gluster/run/test --opt o=bind test

- Create overlay network
   docker network create --driver=overlay --subnet 10.32.0.0/16 --gateway 10.32.0.1 --attachable weave

- Create docker service
docker service create --name=nginx --network weave --mount src=test,dst=/config -e PGID=1000 -e PUID=1000 -p 80:80 -p 443:443 -e TZ=Europe/Berlin lsioarmhf/nginx-aarch64

Diagnostic log
See https://gist.github.com/giminni/1ab53616d6529baeace0f2e6d2eac65a

[fcrisciani]

@giMini can you try: curl -v -4 http://localhost configuration looks good

[deminngi]

@fcrisciani here the result

$curl -v -4 http://localhost

• Rebuilt URL to: http://localhost/
• Trying 127.0.0.1...
• TCP_NODELAY set
• connect to 127.0.0.1 port 80 failed: Connection refused
• Trying 127.0.0.1...
• TCP_NODELAY set
• connect to 127.0.0.1 port 80 failed: Connection refused
• Failed to connect to localhost port 80: Connection refused
• Closing connection 0

[fcrisciani]

@giminni does it work if you do curl inside the container?

[deminngi]

@fcrisciani If I use the docker_gwbridge ip address of the container 172.18.0.3 not 172.18.0.2 it works, I guess something is wrong with the ingress_sbox container or iptables configuration.

$ docker network inspect docker_gwbridge
[
{
"Name": "docker_gwbridge",
"Id": "643d2dc921fc3c5d7a3a7b240f9ae58e49561fc483d57451734fac7546b043a6",
"Created": "2018-07-09T00:45:18.296196724+02:00",
"Scope": "local",
"Driver": "bridge",
"EnableIPv6": false,
"IPAM": {
"Driver": "default",
"Options": {},
"Config": [
{
"Subnet": "172.18.0.0/16",
"Gateway": "172.18.0.1"
}
]
},
"Internal": false,
"Attachable": false,
"Ingress": false,
"ConfigFrom": {
"Network": ""
},
"ConfigOnly": false,
"Containers": {
"e5dc1741e78a1d6ec582a1fb0c91296573006dd7728b391238c4c28c2cfec285": {
"Name": "gateway_e5dc1741e78a",
"EndpointID": "69a4b2a6c5c60114202e71ee191058a52c8b72cfc852b66c9a844835eb9d194b",
"MacAddress": "02:42:ac:12:00:03",
"IPv4Address": "172.18.0.3/16",
"IPv6Address": ""
},
"ingress-sbox": {
"Name": "gateway_ingress-sbox",
"EndpointID": "6619f9c6292791c1dce82cc6972d5d183f2b8542ff6915ea96c50cb321cb31da",
"MacAddress": "02:42:ac:12:00:02",
"IPv4Address": "172.18.0.2/16",
"IPv6Address": ""
}
},
"Options": {
"com.docker.network.bridge.enable_icc": "false",
"com.docker.network.bridge.enable_ip_masquerade": "true",
"com.docker.network.bridge.name": "docker_gwbridge"
},
"Labels": {}
}
]

$curl -v -4 http://172.18.0.3

• Rebuilt URL to: http://172.18.0.3/
• Trying 172.18.0.3...
• TCP_NODELAY set
• Connected to 172.18.0.3 (172.18.0.3) port 80 (#0)

GET / HTTP/1.1
Host: 172.18.0.3
User-Agent: curl/7.58.0
Accept: /

< HTTP/1.1 200 OK
< Server: nginx/1.12.2
< Date: Tue, 10 Jul 2018 16:20:17 GMT
< Content-Type: text/html
< Content-Length: 988
< Last-Modified: Sat, 07 Jul 2018 23:47:45 GMT
< Connection: keep-alive
< ETag: "5b415121-3dc"
< Accept-Ranges: bytes
<


<title>Welcome to our server</title>

[deminngi]

@fcrisciani I can access the vip address from the ingress_sbox shell
$ sudo ip netns exec ingress_sbox sh
# iptables -n -v -L -t mangle
Chain PREROUTING (policy ACCEPT 154 packets, 17089 bytes)
pkts bytes target prot opt in out source destination
0 0 MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 MARK set 0x100
0 0 MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 MARK set 0x100

Chain INPUT (policy ACCEPT 154 packets, 17089 bytes)
pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 174 packets, 13945 bytes)
pkts bytes target prot opt in out source destination
9 815 MARK all -- * * 0.0.0.0/0 10.255.0.12 MARK set 0x100

Chain POSTROUTING (policy ACCEPT 174 packets, 13945 bytes)
pkts bytes target prot opt in out source destination
# curl -4 http://10.255.0.12


<title>Welcome to our server</title>

[fcrisciani]

from the host it matches the DOCKER_INGRESS

Chain PREROUTING (policy ACCEPT 2111 packets, 142K bytes)
 pkts bytes target     prot opt in     out     source               destination
11016  661K DOCKER-INGRESS  all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

and that direct it to the ingress namespace:

Chain DOCKER-INGRESS (2 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 to:172.18.0.2:443
    0     0 DNAT       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 to:172.18.0.2:80

Now as the packets pass through the ingress_sbox they are getting marked:

# nsenter --net=/var/run/docker/netns/ingress_sbox iptables -w1 -n -v -L -t mangle
Chain PREROUTING (policy ACCEPT 8 packets, 672 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:80 MARK set 0x10d
    0     0 MARK       tcp  --  *      *       0.0.0.0/0            0.0.0.0/0            tcp dpt:443 MARK set 0x10d

and then IPVS is load balancing them to the backend container:

# nsenter --net=/var/run/docker/netns/ingress_sbox ipvsadm -l -n
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
FWM  269 rr
  -> 10.255.0.13:0                Masq    1      0          0

Did something changed between your new analysis and the support.sh? I can see that from the support the backend is 10.255.0.13 while in your last dump it is instead 10.255.0.12 and also the firewall marker are not matching, 256 (0x100) instead of 269 (0x10D)

[deminngi]

@fcrisciani
thx for your reply, yes indeed I am doing a lot of things trying to understand why I cannot get the published port from the host.

Why are the packet counter zero in DOCKER-INGRESS and POST-ROUTING?

I will upload an updated docker-support log this morning and reset the packet counter.

BTW is there an issue using kernel 4.4 and iptables 1.6.1?

[deminngi]

@fcrisciani
I uploaded the new diagnostic log and executed w3m http://localhost, got not response, tried http://172.180.0.3 (nginx-1) it was successful

I looked inside the ingress sandbox, the counters are always zero

Here my findings:
$ ip netns exec ingress_sbox sh

# iptables -vnL -t nat
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER_OUTPUT all -- * * 0.0.0.0/0 127.0.0.11
0 0 DNAT icmp -- * * 0.0.0.0/0 10.255.0.35 icmptype 8 to:127.0.0.1

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 DOCKER_POSTROUTING all -- * * 0.0.0.0/0 127.0.0.11
0 0 SNAT all -- * * 0.0.0.0/0 10.255.0.0/16 ipvs to:10.255.0.2

Chain DOCKER_OUTPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 DNAT tcp -- * * 0.0.0.0/0 127.0.0.11 tcp dpt:53 to:127.0.0.11:40192
0 0 DNAT udp -- * * 0.0.0.0/0 127.0.0.11 udp dpt:53 to:127.0.0.11:42917

Chain DOCKER_POSTROUTING (1 references)
pkts bytes target prot opt in out source destination
0 0 SNAT tcp -- * * 127.0.0.11 0.0.0.0/0 tcp spt:40192 to::53
0 0 SNAT udp -- * * 127.0.0.11 0.0.0.0/0 udp spt:42917 to::53

# iptables -vnL -t mangle
Chain PREROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 MARK set 0x103
0 0 MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 MARK set 0x103

Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

Chain OUTPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 10.255.0.35 MARK set 0x103

Chain POSTROUTING (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination

And the endpoint is never reached.
$ ip netns exec ingress_sbox sh
# ipvsadm -l -n --stats
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes
-> RemoteAddress:Port
FWM 259 0 0 0 0 0
-> 10.255.0.36:0 0 0 0 0 0
-> 10.255.0.37:0 0 0 0 0 0

[fcrisciani]

@giminni the default gw network is mainly for north/south communication than for exposing services.
We should focus on the path that I was describing before. trying to see where the packet is not passing.
For the stats what about the host, there is where the translation starts

[deminngi]

I use w3m http://localhost trying to reach port 80
See activity flow from docker host on table nat for chain OUTPUT to DOCKER-INGRESS to POSTROUTING

Chain OUTPUT (policy ACCEPT 1094 packets, 85015 bytes)
pkts bytes target prot opt in out source destination
821 68143 DOCKER-INGRESS all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
0 0 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL

Looking at DOCKER-INGRESS chain got counts on RETURN not for port 80

Chain DOCKER-INGRESS (2 references)
pkts bytes target prot opt in out source destination
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 to:172.18.0.2:443
0 0 DNAT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 to:172.18.0.2:80
1507 116K RETURN all -- * * 0.0.0.0/0 0.0.0.0/0

Going now to ingress sandbox on table mangle I see activity on all chains, but packet counter for port 80 is zero.

Chain PREROUTING (policy ACCEPT 560 packets, 54320 bytes)
pkts bytes target prot opt in out source destination
0 0 MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:80 MARK set 0x100
0 0 MARK tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:443 MARK set 0x100
Chain INPUT (policy ACCEPT 560 packets, 54320 bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 560 packets, 54320 bytes)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 10.255.0.35 MARK set 0x100
Chain POSTROUTING (policy ACCEPT 560 packets, 54320 bytes)
pkts bytes target prot opt in out source destination

Showing the stats inside the sandbox I can't see any activity

IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Conns InPkts OutPkts InBytes OutBytes
-> RemoteAddress:Port
FWM 256 0 0 0 0 0
-> 10.255.0.6:0 0 0 0 0 0
-> 10.255.0.7:0

[fcrisciani]

@giminni I tried the steps that you mentioned at the top on an x86, ubuntu 18.04 instance but I'm not able to reproduce with a vanilla nginx instance, and debugging iptables from github is really tedious.
There is definitely something that is blocking that traffic, you need to do tcpdump on each step and see where does it stops.
So far seem does not even arrive to the bridge in the ingress-sbox. you should be able to enter the namespace: nsenter --net=/var/run/docker/netns/ingress_sbox and do tcpdump on the interface with ip 172.18.0.2. if the traffic does not even arrive there, no point to check further in the chain.

[deminngi]

@fcrisciani thx for instruction
strange is that it works with mode=host.

Can this be a problem with kernel 4.4? and can you show me a working iptables -vnL -t nat on the host and iptables -vnL -t mangle on ingress_sbox?

I am testing using hping3 localhost -p 80 on the host and tcpdump -i eth1 on ingress_sbox. In parallel I use watch -n 2 -d iptables -vnL -t nat without success

UPDATE: If I sent a SYN flag using hping3 -V -S -p 80 localhost it works and the counters are incremented, seeing traffic on eth1 on ingress_sbox.

Why can SYN pass through?

[fcrisciani]

@giminni ingress is exposing TCP ports, ICMP is not going to work.
This is the output of the machine that I tested:
working.txt

This is the tool to get the output: docker run -v /var/run:/var/run --network host --privileged dockereng/network-diagnostic:support.sh > working.txt

Test:

docker network create -d overlay --attachable weave
docker service create --name=nginx --network weave  -p 80:80 -p 443:443 -e TZ=Europe/Berlin nginx

[deminngi]

@fcrisciani I am not using ping I use hping3
Checking port: Here hping3 will send a Syn packet to a specified port (80 in my example). I can control also from which local port will start the scan (30000)

hping3 -V -S -p 80 -s 30000 localhost
Output from hping3 is:
using lo, addr: 127.0.0.1, MTU: 65536
HPING localhost (lo 127.0.0.1): S set, 40 headers + 0 data bytes

on ingress_sbox I got:

IP 172.18.0.1.30000 > 172.18.0.2.http: Flags [S], seq 1083945123, win 512, length 0
IP 172.18.0.2.http > 172.18.0.1.30000: Flags [R.], seq 0, ack 1083945124, win 0, length 0

and I see that SYN is working, but not HTTP

[deminngi]

@fcrisciani
Here my lsmod output:

Module Size Used by
ip_vs_rr 16384 2
xt_ipvs 16384 2
ip_vs 110592 5 ip_vs_rr,xt_ipvs
xt_REDIRECT 16384 1
nf_nat_redirect 16384 1 xt_REDIRECT
xt_nat 16384 13
xt_tcpudp 16384 15
veth 16384 0
vxlan 40960 0
ip6_udp_tunnel 16384 1 vxlan
udp_tunnel 16384 1 vxlan
iptable_mangle 16384 2
xt_mark 16384 3
ipt_MASQUERADE 16384 3
nf_nat_masquerade_ipv4 16384 1 ipt_MASQUERADE
nf_conntrack_netlink 36864 0
iptable_nat 16384 3
nf_conntrack_ipv4 24576 6
nf_defrag_ipv4 16384 1 nf_conntrack_ipv4
nf_nat_ipv4 16384 1 iptable_nat
xt_addrtype 16384 5
iptable_filter 16384 2
xt_conntrack 16384 5
nf_nat 20480 4 nf_nat_redirect,nf_nat_ipv4,xt_nat,nf_nat_masquerade_ipv4
nf_conntrack 126976 7 ip_vs,nf_nat,nf_nat_ipv4,xt_conntrack,nf_nat_masquerade_ipv4,nf_conntrack_netlink,nf_conntrack_ipv4
overlay 45056 1
snd_soc_rockchip_hdmi_dp 16384 0
snd_soc_rockchip_i2s 16384 4
snd_soc_rockchip_spdif 16384 2
rockchip_saradc 16384 0
ip_tables 24576 3 iptable_filter,iptable_mangle,iptable_nat
x_tables 32768 11 xt_ipvs,xt_mark,ip_tables,xt_tcpudp,ipt_MASQUERADE,xt_conntrack,xt_nat,iptable_filter,xt_REDIRECT,iptable_mangle,xt_addrtype
autofs4 40960 0
dw_hdmi_i2s_audio 16384 0