Could you please let me know how to use "ip mptcp end points backup"? #191
Comments
|
Hello, The configuration you did seems good. Did you also configure IP rules and routes? Could you share some captures, one for each interface of the client (or server) to see if the behaviour seems correct? I suspect other subflows to be used, not the one you expect. |
|
Thanks for your support! route on client
routing policy
route on server
I got a following tcpdump at only enp0s9 on client ,but I couldn't get capture at enp0s8.
I send a file(200M) from client to server and I set 192.168.30.2 as target IP and I use only this Do I need other configuration? or Is there some incorrect configuration? |
|
@matttbe |
|
Have you tried executing the command |
|
@VenkateswaranJ @matttbe enp0s9 on client
enp0s8 on clinet
Do you have any idea to make a backup path? |
|
Hi @Kenoko Sorry, I was not available these last days (and I still have a lot of stuff to catch up). Routes seem good. I noticed that the command you sent was wrong: Just to be sure, can you give me the output of this command: If you see the backup flag, can you also try to set the backup flag on the server side as well and take traces again? Just to check if there is an issue only with the client or also with server. |
|
@matttbe Thank you for your reply despite your busy schedule.
Server
However I couldn't see backup flag.
Server
tcp dump result is as below. There is no backup flag also.
Do I need to change something on configure of my servers? |
|
Strange, I will try to investigate more. In theory, we should configure the client with: (subflow is required) And the server: I guess the order of the flags there should not change anything because all flags are sent to the kernel and read when needed but I will investigate. |
|
@matttbe I executed client_setting.txt According to "RFC8684",It is written as follows. so I think there is no traffic on enp0s9 while enp0s8 is available.
|
|
@Kenoko when discussing about this bug at the last meeting, we were wondering if this issue has not been fixed in a more recent kernel. We remembered having fixed something around there. By chance, could you try with a more recent kernel, e.g. 5.12? |
|
Thank you for your advice and support.I installed following kernels on client/server.
I matched the kernel version between client and server and tried to make backup path. However the result was same as before. the result is as below.
Could you please check this? |
|
Thank you for having checked that!
That's already better! But if you see traffic over this interface while non backup path are still active, there is an issue on the packet scheduler side.
Mmh, maybe a similar issue in the PM is there for the server side if you don't see the backup flag in the SYN+ACK+MP_JOIN.
Yes but to be honest with you, it might take some time for me to catch-up my backlog :) |
|
@matttbe |
|
We quickly discussed about this issue at the last weekly meeting we have and something that should be checked is, when the backup flag is correctly set in the SYN, maybe the traffic we can see on this interface is only "reinjected" traffic. If the non backup link is lossy, we can have a lot of reinjections, see #177. Is the non backup link lossy in your case? Is the backup link "fully used": is the whole the capacity of the link used? |
|
Thank you for your checking. |
|
I will need to check this in more details but with a simple setup, I can see a lot of traffic on the backup link: Setup: Utilisation: Still, it looks like the backup link is only used to upload data of my bi-directional flow.
Be careful that here, we don't talk about TCP re-transmission but MPTCP re-injection: packets that were sent in one subflow but reinjected to another one. |
[...]
The problem is that when the 'backup' flag is set, the flag is announced to the peer (that is: enforced in the opposite direction), but not set in the forward/egress direction. That is actually compliant with the RFC - the backup flag is a per direction feature - but very confusing. There is a simple fix for this - setting the backup flag in both direction - but it will break active backup (when no user-space PM is in action). A possibly complete solution would be to additionally introduce PM logic inside the kernel to automatically close a subflow making no forward progresses and triggerring too much re-injection when other subflows are available, as per paragraph 3.3.6. in the RFC. |
If we tell the PM an endpoint has to be considered as backup, should it eventually ensure that the packet scheduler is also taking this into consideration even if the other peer didn't set the backup flag? It is true that the backup flag is there to tell the other peer a path should be used as a backup one and it should use it only if there is no other non backup path available (still, not a MUST). But typically, the client knows a path is more costly and might mark it as backup for the other peer but also for itself, not to use it if other paths are available, no? :)
I would need to check again but I think in my case, the backup flag was set on both the SYN and SYN+ACK. |
Right, the server is currently mirroring back the backup flag from ingress syn MPJ packet into the SYN-ACK reply. But the client code is ignoring such bit - which is a bug, to be fixed. The discussion about active backup still applies - when fixing such bug we additionally need a smarter in kernel PM. |
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: multipath-tcp/mptcp_net-next#191 Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: multipath-tcp/mptcp_net-next#191 Signed-off-by: Paolo Abeni <[email protected]> Signed-off-by: Mat Martineau <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: #191 Reviewed-by: Mat Martineau <[email protected]> Signed-off-by: Paolo Abeni <[email protected]>
the parsed incoming backup flag is not propagated to the subflow itself, the client may end-up using it to send data. Closes: multipath-tcp/mptcp_net-next#191 Signed-off-by: Paolo Abeni <[email protected]> Signed-off-by: Mat Martineau <[email protected]> Signed-off-by: David S. Miller <[email protected]>
Add test to validate BPF verifier's register range bounds tracking logic. The main bulk is a lot of auto-generated tests based on a small set of seed values for lower and upper 32 bits of full 64-bit values. Currently we validate only range vs const comparisons, but the idea is to start validating range over range comparisons in subsequent patch set. When setting up initial register ranges we treat registers as one of u64/s64/u32/s32 numeric types, and then independently perform conditional comparisons based on a potentially different u64/s64/u32/s32 types. This tests lots of tricky cases of deriving bounds information across different numeric domains. Given there are lots of auto-generated cases, we guard them behind SLOW_TESTS=1 envvar requirement, and skip them altogether otherwise. With current full set of upper/lower seed value, all supported comparison operators and all the combinations of u64/s64/u32/s32 number domains, we get about 7.7 million tests, which run in about 35 minutes on my local qemu instance without parallelization. But we also split those tests by init/cond numeric types, which allows to rely on test_progs's parallelization of tests with `-j` option, getting run time down to about 5 minutes on 8 cores. It's still something that shouldn't be run during normal test_progs run. But we can run it a reasonable time, and so perhaps a nightly CI test run (once we have it) would be a good option for this. We also add a small set of tricky conditions that came up during development and triggered various bugs or corner cases in either selftest's reimplementation of range bounds logic or in verifier's logic itself. These are fast enough to be run as part of normal test_progs test run and are great for a quick sanity checking. Let's take a look at test output to understand what's going on: $ sudo ./test_progs -t reg_bounds_crafted #191/1 reg_bounds_crafted/(u64)[0; 0xffffffff] (u64)< 0:OK ... #191/115 reg_bounds_crafted/(u64)[0; 0x17fffffff] (s32)< 0:OK ... #191/137 reg_bounds_crafted/(u64)[0xffffffff; 0x100000000] (u64)== 0:OK Each test case is uniquely and fully described by this generated string. E.g.: "(u64)[0; 0x17fffffff] (s32)< 0". This means that we initialize a register (R6) in such a way that verifier knows that it can have a value in [(u64)0; (u64)0x17fffffff] range. Another register (R7) is also set up as u64, but this time a constant (zero in this case). They then are compared using 32-bit signed < operation. Resulting TRUE/FALSE branches are evaluated (including cases where it's known that one of the branches will never be taken, in which case we validate that verifier also determines this as a dead code). Test validates that verifier's final register state matches expected state based on selftest's own reg_state logic, implemented from scratch for cross-checking purposes. These test names can be conveniently used for further debugging, and if -vv verboseness is requested we can get a corresponding verifier log (with mark_precise logs filtered out as irrelevant and distracting). Example below is slightly redacted for brevity, omitting irrelevant register output in some places, marked with [...]. $ sudo ./test_progs -a 'reg_bounds_crafted/(u32)[0; U32_MAX] (s32)< -1' -vv ... VERIFIER LOG: ======================== func#0 @0 0: R1=ctx(off=0,imm=0) R10=fp0 0: (05) goto pc+2 3: (85) call bpf_get_current_pid_tgid#14 ; R0_w=scalar() 4: (bc) w6 = w0 ; R0_w=scalar() R6_w=scalar(smin=0,smax=umax=4294967295,var_off=(0x0; 0xffffffff)) 5: (85) call bpf_get_current_pid_tgid#14 ; R0_w=scalar() 6: (bc) w7 = w0 ; R0_w=scalar() R7_w=scalar(smin=0,smax=umax=4294967295,var_off=(0x0; 0xffffffff)) 7: (b4) w1 = 0 ; R1_w=0 8: (b4) w2 = -1 ; R2=4294967295 9: (ae) if w6 < w1 goto pc-9 9: R1=0 R6=scalar(smin=0,smax=umax=4294967295,var_off=(0x0; 0xffffffff)) 10: (2e) if w6 > w2 goto pc-10 10: R2=4294967295 R6=scalar(smin=0,smax=umax=4294967295,var_off=(0x0; 0xffffffff)) 11: (b4) w1 = -1 ; R1_w=4294967295 12: (b4) w2 = -1 ; R2_w=4294967295 13: (ae) if w7 < w1 goto pc-13 ; R1_w=4294967295 R7=4294967295 14: (2e) if w7 > w2 goto pc-14 14: R2_w=4294967295 R7=4294967295 15: (bc) w0 = w6 ; [...] R6=scalar(id=1,smin=0,smax=umax=4294967295,var_off=(0x0; 0xffffffff)) 16: (bc) w0 = w7 ; [...] R7=4294967295 17: (ce) if w6 s< w7 goto pc+3 ; R6=scalar(id=1,smin=0,smax=umax=4294967295,smin32=-1,var_off=(0x0; 0xffffffff)) R7=4294967295 18: (bc) w0 = w6 ; [...] R6=scalar(id=1,smin=0,smax=umax=4294967295,smin32=-1,var_off=(0x0; 0xffffffff)) 19: (bc) w0 = w7 ; [...] R7=4294967295 20: (95) exit from 17 to 21: [...] 21: (bc) w0 = w6 ; [...] R6=scalar(id=1,smin=umin=umin32=2147483648,smax=umax=umax32=4294967294,smax32=-2,var_off=(0x80000000; 0x7fffffff)) 22: (bc) w0 = w7 ; [...] R7=4294967295 23: (95) exit from 13 to 1: [...] 1: [...] 1: (b7) r0 = 0 ; R0_w=0 2: (95) exit processed 24 insns (limit 1000000) max_states_per_insn 0 total_states 2 peak_states 2 mark_read 1 ===================== Verifier log above is for `(u32)[0; U32_MAX] (s32)< -1` use cases, where u32 range is used for initialization, followed by signed < operator. Note how we use w6/w7 in this case for register initialization (it would be R6/R7 for 64-bit types) and then `if w6 s< w7` for comparison at instruction #17. It will be `if R6 < R7` for 64-bit unsigned comparison. Above example gives a good impression of the overall structure of a BPF programs generated for reg_bounds tests. In the future, this "framework" can be extended to test not just conditional jumps, but also arithmetic operations. Adding randomized testing is another possibility. Some implementation notes. We basically have our own generics-like operations on numbers, where all the numbers are stored in u64, but how they are interpreted is passed as runtime argument enum num_t. Further, `struct range` represents a bounds range, and those are collected together into a minimal `struct reg_state`, which collects range bounds across all four numberical domains: u64, s64, u32, s64. Based on these primitives and `enum op` representing possible conditional operation (<, <=, >, >=, ==, !=), there is a set of generic helpers to perform "range arithmetics", which is used to maintain struct reg_state. We simulate what verifier will do for reg bounds of R6 and R7 registers using these range and reg_state primitives. Simulated information is used to determine branch taken conclusion and expected exact register state across all four number domains. Implementation of "range arithmetics" is more generic than what verifier is currently performing: it allows range over range comparisons and adjustments. This is the intended end goal of this patch set overall and verifier logic is enhanced in subsequent patches in this series to handle range vs range operations, at which point selftests are extended to validate these conditions as well. For now it's range vs const cases only. Note that tests are split into multiple groups by their numeric types for initialization of ranges and for comparison operation. This allows to use test_progs's -j parallelization to speed up tests, as we now have 16 groups of parallel running tests. Overall reduction of running time that allows is pretty good, we go down from more than 30 minutes to slightly less than 5 minutes running time. Acked-by: Eduard Zingerman <[email protected]> Signed-off-by: Andrii Nakryiko <[email protected]> Acked-by: Shung-Hsi Yu <[email protected]> Link: https://lore.kernel.org/r/[email protected] Signed-off-by: Alexei Starovoitov <[email protected]>
I try to make backup-interface, but I cannot do it,
Server and Client OS: Ubuntu21.04(kernel:5.11.0-16-generic)
I confirmed I could send mptcp packets on both paths from client to server. so I want to make an active/standby structure in a next step.
I executed a following command on client side, however the interface didn't become backup mode.The mptcp traffic kept running.
ip mptcp add endpoint 192.168.20.2 dev enp0s9 backupand I created signal endpoint at server as below.
ip mptcp add endpoint 192.168.40.2 dev enp0s9 signalThe limit setting is
ip mptcp limits set subflow 1 add_addr_accepted 1on both client and server.So Could you please let me know how to make backup interface?
I could make it by using mptcp v0 while I referred a following site, so I want to make it with mptcpv1 also.
http://multipath-tcp.org/pmwiki.php/Users/Tools
The text was updated successfully, but these errors were encountered: