MPTCP issue with simultaneous interface existence

[webgenius-coder]

I've been able to port MPTCP v0.92 successfully on Android. I had to make a minor change in framework to retain both the wlan0 and rmnet_data2 interfaces.

However when I check the packet capture after downloading a file from MPTCP capable site, I see the device is using only wlan0 interface always. When wlan0 is turned off, rment_data2 interface is used. Both the interfaces are not able to send packets simultaneously.

As per the instructions shared in MPTCP site, routing rule and routes are configured as follows:
wlan0 IP: 172.26.14.202
rmnet_data2 IP: 100.95.128.56

Device# ip route show
default via 172.26.14.1 dev wlan0
100.95.128.48/28 dev rmnet_data2 proto kernel scope link src 100.95.128.56
172.26.14.0/24 dev wlan0 proto kernel scope link src 172.26.14.202

Device# ip route show table 1
default via 172.26.14.1 dev wlan0
172.26.14.0/24 dev wlan0 scope link

Device# ip route show table 2
100.95.128.48/28 dev rmnet_data2 scope link

Device# ip rule show
0: from all lookup local
9998: from 100.95.128.56 lookup 2
9999: from 172.26.14.202 lookup 1
10000: from all fwmark 0xc0000/0xd0000 lookup legacy_system
10500: from all oif dummy0 uidrange 0-0 lookup dummy0
10500: from all oif rmnet_data0 uidrange 0-0 lookup rmnet_data0
10500: from all oif wlan0 uidrange 0-0 lookup wlan0
10500: from all oif r_rmnet_data0 uidrange 0-0 lookup r_rmnet_data0
10500: from all fwmark 0xc0000/0xc0000 oif rmnet_data1 uidrange 0-0 lookup rmnet_data1
10500: from all fwmark 0x40000/0x40000 oif rmnet_data2 uidrange 0-0 lookup rmnet_data2
13000: from all fwmark 0x10063/0x1ffff lookup local_network
13000: from all fwmark 0x1000b/0x1ffff lookup rmnet_data0
13000: from all fwmark 0x10064/0x1ffff lookup wlan0
13000: from all fwmark 0x1000a/0x1ffff lookup r_rmnet_data0
13000: from all fwmark 0xd0065/0xdffff lookup rmnet_data1
13000: from all fwmark 0x50066/0x5ffff lookup rmnet_data2
14000: from all oif dummy0 lookup dummy0
14000: from all oif rmnet_data0 lookup rmnet_data0
14000: from all oif wlan0 lookup wlan0
14000: from all oif r_rmnet_data0 lookup r_rmnet_data0
14000: from all fwmark 0xc0000/0xc0000 oif rmnet_data1 lookup rmnet_data1
14000: from all fwmark 0x40000/0x40000 oif rmnet_data2 lookup rmnet_data2
15000: from all fwmark 0x0/0x10000 lookup legacy_system
16000: from all fwmark 0x0/0x10000 lookup legacy_network
17000: from all fwmark 0x0/0x10000 lookup local_network
18000: from all iif r_rmnet_data0 lookup wlan0
18000: from all iif r_rmnet_data0 lookup dummy0
19000: from all fwmark 0xb/0x1ffff lookup rmnet_data0
19000: from all fwmark 0x64/0x1ffff lookup wlan0
19000: from all fwmark 0xa/0x1ffff lookup r_rmnet_data0
22000: from all fwmark 0x0/0xffff lookup wlan0
23000: from all fwmark 0x0/0xffff uidrange 0-0 lookup main
32000: from all unreachable

I captured the dmesg with one onterface only, and second time with both the interfaces up. When both the interfaces are up. I see the following print which was not seen when only one interface was up:

android/kernel/msm-4.4/net/mptcp/mptcp_ctrl.c: mptcp_alloc_mpcb: created mpcb with token 0xb97602f8
android/kernel/msm-4.4/net/mptcp/mptcp_ctrl.c: mptcp_add_sock: token 0xb97602f8 pi 1, src_addr:172.26.14.202:45021 dst_addr:130.104.228.140:80, cnt_subflows now 1
09-10 17:01:41.121 0 0 F : mptcp default window
09-10 17:01:41.228 0 0 F : mptcp tcp_clean_rtx_queue
09-10 17:01:43.337 0 0 F : mptcp tcp_clean_rtx_queue
09-10 17:01:43.337 0 0 F : mptcp tcp_clean_rtx_queue
android/kernel/msm-4.4/net/mptcp/mptcp_ctrl.c: mptcp_del_sock: Removing subsock tok 0xaa95d416 pi:1 state 7 is_meta? 0
android/kernel/msm-4.4/net/mptcp/mptcp_ctrl.c: mptcp_sock_destruct destroying meta-sk

Would really appreciate input on this. I'm not sure whether the issue is because of IP route and rule configuration, or with the subflows creation.

[cpaasch]

Backports of such old releases are hard to support.

You could try to see if MPTCP has correctly been negotiated,...

[matttbe]

A while ago, with old Android devices, we had to force the system to allow traffic on both interfaces: https://github.com/MPTCP-smartphone-thesis/MultipathControl

With new version of Android, I think there is an option in the developer menu of Android (Mobile data always active?) that was enough to allow this.

Also, make sure SELinux is disabled, there are apps for that.

[ManuAryan]

I have also observed one thing that mptcp_fullmesh module is not loaded and also there is no proc file created , net.mptcp.mptcp_path_manager shows as default . Is it the reason why no subflows are being created for the rmnet_data2 IP ? netstat entry also shows TCP link is established only for wlan0 but no entry is found for rmnet_data2 interface .

[cpaasch]

Yes, you need to make sure that fullmesh is compiled in the kernel and that you set net.mptcp.mptcp_path_manager as fullmesh.

[ManuAryan]

After making sure the full_mesh module loads below is the snapshot of our device

Device:/ # netstat
Active Internet connections (w/o servers)
Proto Recv-Q Send-Q Local Address Foreign Address State
tcp 0 0 172.26.14.202:41044 130.104.228.140:http ESTABLISHED
tcp 0 0 172.26.14.202:41042 130.104.228.140:http ESTABLISHED
tcp 0 0 172.26.14.202:41048 130.104.228.140:http ESTABLISHED
tcp 0 0 172.26.14.202:45049 maa05s09-in-f3.1e:https ESTABLISHED
tcp6 0 0 ::ffff:172.26.14.:44802 bom07s18-in-f10.1:https ESTABLISHED
tcp6 0 0 ::ffff:100.86.119:40811 sb-in-f188.1e100.n:5228 ESTABLISHED
tcp6 0 0 ::ffff:172.26.14.:46133 sb-in-f188.1e100.n:5228 ESTABLISHED

> Device:/proc/net/mptcp_net # cat mptcp

sl loc_tok rem_tok v6 local_address remote_address st ns tx_queue rx_queue inode
0: 34415CA4 CF4030E0 0 CA0E1AAC:A058 8CE46882:0050 01 01 00000000:00000000 109147
1: 1BD26259 F54664C 0 CA0E1AAC:A054 8CE46882:0050 01 01 00000000:00000000 107924
2: D1741322 4C8FAB6F 0 CA0E1AAC:A052 8CE46882:0050 01 01 00000000:00000000 105425

> Device:/proc/net # cat mptcp_fullmesh

Index, Address-ID, Backup, IP-address
IPv4, next v4-index: 5
3, 4, 0, 100.86.119.57
4, 5, 0, 172.26.14.202
IPv6, next v6-index: 1
0, 8, 0, 2401:4900:2119:777b:8599:c22e:4bf8:412c

> Device:/sys/module/mptcp_fullmesh/parameters # cat num_subflows

3
> Device:/sys/module/mptcp_fullmesh/parameters # cat create_on_err
1

> Device:/ # sysctl net.mptcp

net.mptcp.mptcp_checksum = 0
net.mptcp.mptcp_debug = 1
net.mptcp.mptcp_enabled = 1
net.mptcp.mptcp_path_manager = fullmesh
net.mptcp.mptcp_scheduler = default
net.mptcp.mptcp_syn_retries = 3
net.mptcp.mptcp_version = 0

The issue we are facing now here is , tcplink for another rmnet_data2 IP address 100.86.119.57 is not getting established in netstat,that is the only deviation we are observing when compared with our working device nexus 5. Is it the reason why another subflow is not getting created ? please provide any input on how to debug further.

> we are also seeing mptcp_add_sock is creating only for wlan0 IP but the same is not happening for rmnet_data 2 ip

mptcp_add_sock: token 0x1bd26259 pi 1, src_addr:172.26.14.202:41044 dst_addr:130.104.228.140:80, cnt_subflows now 1

wlan0 : 172.26.14.202
rmnet_data2:100.86.119.57

[ManuAryan]

I am unable to change the path_manager using sysctl , Is there any other way to set it in kernel code in mptcp v0.92 ?

[cpaasch]

What error message do you get?

You can also configure the default path manager in the kernel configuration to fullmesh when you are compiling your kernel.

[ManuAryan]

But full mesh is written as loadable kernel module, full_mesh..c has module_init(register function).
how can I make it built-in kernel module ?
Right now after insmoding the full_mesh.ko I am facing the issue while creating mptcp4_mini_subsocket with permission error as follows
/mptcp_ipv4.c: mptcp_init4_subsockets inet_create failed ret: -13

since it doesn't have AID_INET or CAP_NET_RAW permission which is checked as part of CONFIG_ANDROID_PARANOID_NETWORK in android

[matttbe]

how can I make it built-in kernel module ?

there are many ways. Can you launch make menuconfig and update the defconfig file used to compile the kernel? Just need to switch from =m to =y

[dhawaskar]

I am stuck with the same problem.

I have ported v.90 MPTCP onto Android phone. I only see one single flow created all time instead of two.
Single Sub Flow used Wifi always instead of Cellular and when wifi is disabled, cellular is not used and vice versa.

I followed the link http://amiusingmptcp.de/#ports and it does show green yes mark and only single sub flow. I have setup the MPTCP as below:
marlin:/data/local/tmp # sysctl net.mptcp
net.mptcp.mptcp_checksum = 1
net.mptcp.mptcp_debug = 0
net.mptcp.mptcp_enabled = 1
net.mptcp.mptcp_path_manager = fullmesh
net.mptcp.mptcp_scheduler = default
net.mptcp.mptcp_syn_retries = 3
marlin:/data/local/tmp #

Both IP address are listed in mptcp_fullmesh as well:
marlin:/data/local/tmp # cd /proc/net/
marlin:/proc/net # cat mptcp_fullmesh
Index, Address-ID, Backup, IP-address
IPv4, next v4-index: 3
1, 2, 0, 10.222.126.90
2, 3, 0, 192.168.5.111
IPv6, next v6-index: 0
marlin:/proc/net #

I tried changing the num_subflow in sys/modules/mptcp_fullmesh/parameters/sub_flow to 2 but no luck.

Any help or directions would be greatly appreciated.

[matttbe]

Also, make sure SELinux is disabled, there are apps for that.

Did you check that?

[ManuAryan]

If the device is failing to create subsocket with the permission denied error issue
/mptcp_ipv4.c: mptcp_init4_subsockets inet_create failed ret: -13
then it is due to the CONFIG_ANDROID_PARANOID_NETWORK macro which introduced in the android kernel which allows creating the socket only for certain user groups ,you can comment out this macro part in net/ipv4/af_inet.c file and check once.
It worked for me and I am able to create multiple subflows.

[dhawaskar]

After setting selinux to permissive I can see 3 sub flows being created but, it never uses cellular rmnet_data0. All are created on the WiFi interface.

Any guideline on this would be really helpful.

[matttbe]

Did you configure the routing? http://multipath-tcp.org/pmwiki.php/Users/ConfigureRouting

[ManuAryan]

Yes I did,
Instead of table 1 and table 2, I made use of already created table for rmnet_data2 and wlan0
ip rule add from 10.1.1.2 table wlan0
ip rule add from 10.1.2.2 table rmnet_data2

[dhawaskar]

Thank you so much with all the help.
After setting the selinux to Permissive by doing setenforce 0 and setting up routing table as given in http://multipath-tcp.org/pmwiki.php/Users/ConfigureRouting. Now I can create 6 sub-flows using both cellular and WiFi. Thank you so much for the help. I wish I had asked earlier instead of surfing web for almost a week to find the answers.

[dhawaskar]

Yes I did,
Instead of table 1 and table 2, I made use of already created table for rmnet_data2 and wlan0
ip rule add from 10.1.1.2 table wlan0
ip rule add from 10.1.2.2 table rmnet_data2

Hi,
I had lot of issues but now it is all fixed and I can see 6 flows being created. I followed these steps. Hope it helps.
getenforce: This will be set to enforcing initially.
Do “setenforce 0” and “getneforce” you will see “Permissive”
Next set the routing table as given below. You need to change the IP and interface name based on the device.
Setting routing table:http://multipath-tcp.org/pmwiki.php/Users/ConfigureRouting
ip rule add from 192.168.5.111 table 1
ip rule add from 10.121.181.106 table 2

Configure the two different routing tables

ip route add 192.168.5.0/24 dev wlan0 scope link table 1
ip route add default via 192.168.5.1 dev wlan0 table 1

ip route add 10.121.181.104/30 dev rmnet_data0 scope link table 2
ip route add default via 10.121.181.105 dev rmnet_data0 table 2

default route for the selection process of normal internet-traffic

ip route add default scope global via 10.121.181.105 dev rmnet_data0
Another important setting is your MPTCP configuration:
Path manager should always be fullmesh else new sub-flows will never be created.
Set the below setting in your device:
marlin:/ # sysctl net.mptcp
net.mptcp.mptcp_checksum = 1
net.mptcp.mptcp_debug = 1 -> Helps to debug the MPTCP
net.mptcp.mptcp_enabled = 1
net.mptcp.mptcp_path_manager = fullmesh
net.mptcp.mptcp_scheduler = default
net.mptcp.mptcp_syn_retries = 3
marlin:/ #
When I had a problem I asked question in the forum and its link: #360
Then visit http://amiusingmptcp.de/ from the phone to see how flows are created: In my experiments there were 6 flows created.