Subflows won't be used when the interface for the main flow is a subinterface

[arinc9]

The default route is via 192.168.13.1 dev enp3s0.3.

I've set up the subflow I want to use:

$ sudo ip mp e
192.168.14.2 id 2 subflow dev enp3s0.4 

Then described the gateway for the subflow path:

sudo ip rule add from 192.168.14.2 table 100
sudo ip route add default via 192.168.14.1 table 100

The subflow won't be used:

$ mptcpize run iperf3 -c hostIPv4
[       CREATED] token=20b1293a remid=0 locid=0 saddr4=192.168.13.2 daddr4=hostIPv4 sport=58362 dport=5201
Connecting to host hostIPv4, port 5201
[       CREATED] token=973e67fc remid=0 locid=0 saddr4=192.168.13.2 daddr4=hostIPv4 sport=58366 dport=5201
[  5] local 192.168.13.2 port 58366 connected to hostIPv4 port 5201

enp3s0.3 and enp3s0.4 are VLAN interfaces so they are enp3s0.3@enp3s0 and enp3s0.4@enp3s0. I've noticed that if the default route is a subinterface, subflows won't be created. I've also experienced this issue with DSA interfaces hence my belief that specifically subinterfaces, not just VLAN interfaces, are affected.

[asirir]

Hi,

I wonder if you need to set src on your routing table entries? Like:

hostIPv4 via 192.168.13.1 dev enp3s0.3 src <source_ip>

My setup is different (not fullmesh) but I do set the source IP to make the routing more specific.

[arinc9]

I don't think src is necessary as there is only one IPv4 address assigned to enp3s0.3. I've tested it anyway; it doesn't change the current behaviour.

sudo ip route del hostIPv4 via 192.168.13.1 dev enp3s0.3
sudo ip route add hostIPv4 via 192.168.13.1 dev enp3s0.3 src 192.168.13.2

[asirir]

What's the mptcp configuration on hostIPv4? (I'm assuming the setup you describe in the question is just for the host from which you are running the iperf3 command)

[arinc9]

The host acting as an iperf3 server has net.mptcp.enabled=1 and mptcpize run iperf3 -s is running. No mptcp endpoint is configured. The behaviour is the same if the default route is described as a signal path:

$ sudo ip mp e                                  
hostIPv4 id 1 signal dev eth0 

Both hosts run Ubuntu's 6.8.0-35-generic kernel.

[asirir]

Hi,

How about the MPTCP limits on each host? that is, ip mptcp limits show - I don't think you need a signal endpoint on the remote host in this case, but I think you need at least subflow 1 add_addr_accepted 0 setup on both ends to make this work.

Also, do you need fullmesh endpoints on your source server? If you only need an additional subflow betwen the two hosts, a subflow endpoint would do no?

[arinc9]

The limits are fine. I can see that subflows are created if the path of the default route is utilised. So I don't think this is a case of improper configuration.

$ mptcpize run iperf3 -c hostIPv4
[       CREATED] token=7921b290 remid=0 locid=0 saddr4=192.168.1.2 daddr4=hostIPv4 sport=52718 dport=5201
[   ESTABLISHED] token=7921b290 remid=0 locid=0 saddr4=192.168.1.2 daddr4=hostIPv4 sport=52718 dport=5201
Connecting to host hostIPv4, port 5201
[       CREATED] token=0b497c8d remid=0 locid=0 saddr4=192.168.1.2 daddr4=hostIPv4 sport=52728 dport=5201
[  5] local 192.168.1.2 port 52728 connected to hostIPv4 port 5201
[   ESTABLISHED] token=0b497c8d remid=0 locid=0 saddr4=192.168.1.2 daddr4=hostIPv4 sport=52728 dport=5201
[SF_ESTABLISHED] token=0b497c8d remid=0 locid=1 saddr4=192.168.13.2 daddr4=hostIPv4 sport=40195 dport=5201 backup=0 ifindex=11
[SF_ESTABLISHED] token=7921b290 remid=0 locid=1 saddr4=192.168.13.2 daddr4=hostIPv4 sport=35617 dport=5201 backup=0 ifindex=11
[SF_ESTABLISHED] token=7921b290 remid=0 locid=2 saddr4=192.168.14.2 daddr4=hostIPv4 sport=32779 dport=5201 backup=0 ifindex=12
[SF_ESTABLISHED] token=0b497c8d remid=0 locid=2 saddr4=192.168.14.2 daddr4=hostIPv4 sport=60581 dport=5201 backup=0 ifindex=12

The server only needs to use a single endpoint so it is a half mesh topology. To that extend, I've taken away the fullmesh flag on both subflows:

$ sudo ip mp e
192.168.13.2 id 1 subflow dev enp3s0.3 
192.168.14.2 id 2 subflow dev enp3s0.4 

The behaviour is the same.

[arinc9]

I think I'm onto something. I've been struggling with making MPTCP create subflows on a system without playing around with the routes. Limits are fine on both sides. I've noticed that subflows won't be created if the interface for the main flow is a subinterface, meaning it's a user of another interface. lan1@eth0, a DSA interface; enp3s0.3@enp3s0, a VLAN interface. I'll investigate further.

Edit: I've updated the main comment with my findings.

[asirir]

Tbh, I have a similar setup - multiple sub interfaces on the main interfaces (carrying different subnets). I don't have any vlan setup. I am able to establish subflows between just the sub-interfaces (with the primary flow also on the other sub-interface).

[arinc9]

I don't use alias interfaces so I can't make a comment. There clearly is something wrong with MPTCP when using VLAN and DSA interfaces.

[pabeni]

Note that the mptcp monitor shows the lack of MPTCP_EVENT_ESTABLISHED events, thus the mptcp connection will not try to establish additional subflows.

Can you observe the same behavior on top of the current 'net' or 'net-next' tree?

The attached tentative reproducer works fine for me on a somewhat recent 'net-next'.
repro2.sh.txt (I had to append a '.txt' suffix to please the github infra).

I guess 6.8 is lacking some fixes/improvement.

Could you also please attach a pcap captures with the first few packets of the established connection?

[arinc9]

I will test this soon. Thanks Paolo.

[matttbe]

The default route is via 192.168.13.1 dev enp3s0.3.

I've set up the subflow I want to use:

$ sudo ip mp e
192.168.14.2 id 2 subflow dev enp3s0.4

Be careful that when you specify the device when adding the endpoint, the subflow will be forced to use this outgoing interface ...

Then described the gateway for the subflow path:

sudo ip rule add from 192.168.14.2 table 100
sudo ip route add default via 192.168.14.1 table 100

... In this case, the rule you added is not going to be used by the subflow I think, because oif is missing.

You can check your routing rules with ip route get <destIP> from <srcIP> oif <iface> (required IPRoute2, not the one from Busybox), e.g.

ip route get 1.1.1.1 from 192.168.14.2 oif enp3s0.4

[matttbe]

In other words, if you added the correct interface with the endpoint, you should not need to add these rules. That's what is mentioned in the 'warning' box: https://www.mptcp.dev/pm.html