Fix VPN traffice not displayed bug

[zhangxp1998]

Fix issue #126

[jconley]

tested and am now seeing traffic on the vpn utun interface

[zhangxp1998]

Minor change, merging now

[imsnif]

@zhangxp1998 - great fix! Thanks for taking care of this so quickly. Do you think this only happens on macos?

[zhangxp1998]

@zhangxp1998 - great fix! Thanks for taking care of this so quickly. Do you think this only happens on macos?

It happens to all platforms which libpnet uses bpf. Which means... FreeBSD as well?

https://github.com/libpnet/libpnet/blob/651e362f464f5891f94fdd317b0dd2189e7173fa/pnet_datalink/src/lib.rs#L54

Maybe we should get a FreeBSD box and test.

[zhangxp1998]

I'm considering working on libpnet's BPF code to insert a fake header. That way we will solve this once for all.

[imsnif]

That's cool - I think they'd also be happy with more hands maintaining the repo if you feel like it.

[thepacketgeek]

@zhangxp1998 Do you know if libpnet changed with the point_to_point header? I'm currently not seeing utun traffic unless I revert this change. It seems the offset is only needed for loopbacks on macos now?

[jconley]

I am having the same issue - bandwhich has not worked since I upgraded to a newer NordVPN client and started using their NordLynx/Wireguard protocol. Not sure if this is related to it not being a macOS built-in protocol... or their own implementation at that. Setting the payload_offset back to zero works.

[thepacketgeek]

@jconley That's helpful context, the utun interface I tested with is also Wireguard (using the App Store Wireguard client)

[imsnif]

Thanks for weighing in on this, @thepacketgeek and @jconley. IMO the best thing to do right now would be to hide this functionality behind a command line flag (unless you have other suggestions?). Would any of you like to work on this?

[thepacketgeek]

@imsnif I'm checking into it now, trying to find out if it's just a WireGuard thing (which would make for a good CLI flag), or changed behavior across utun interfaces with libpnet.

[zhangxp1998]

I guess this is because wireguard changed how their network interface work?

[thepacketgeek]

@zhangxp1998 Yes, with some testing it does seem to be Wireguard specific. Let me see if there's something bandwhich can detect automatically, otherwise it might make sense to have a --wireguard flag to bypass the offset code (which is still required for loopback & non-wg utuns) :)

[jconley]

@thepacketgeek i actually configured the app-store NordVPN client to use IKEv2 - and only payload_offset=0 works, so it does not seem exclusive to Wireguard. I notice with NordVPN IKEv2 enabled, it is not using the built-in IKEv2 profile from macOS - but a profile that routes it through the NordVPN process, just like the Wireguard configuration. for reference, the older non-app-store NordVPN client would actually setup a native IKEv2 VPN profile, and that was the last time it worked with payload_offset=14. so it would seem the differentiator (so far) would be whether traffic is routed through the 3rd party VPN process, or whether the OS handles it with a built-in implementation.

[thepacketgeek]

@jconley Oh, super helpful! So I'm thinking of leaving the current behavior as default and providing something like a --no-payload-offset option that sets payload_offset = 0 (With help text that this may be needed for some tunnel interfaces)? Thoughts @imsnif ?

[TheLostLambda]

Heya! Just swooping in because I have a private Wireguard network bridging a bunch of my devices. When configured with just wg-quick and opening a tunnel to another machine I own, bandwhich (the latest main branch) seems to pick up on all of the data being transferred. Moving a couple hundred MB over sftp registers the correct size in cumulative mode -t.

I think this backs up @jconley 's theory that it's something other than Wireguard itself causing the issue here :)

[jconley]

@TheLostLambda can you verify if it's counting only the encrypted traffic to the vpn server... or also the functional non-vpn-encrypted data to the sftp host. bandwhich can pick up one or both, but the latter is the useful info that has been missing. i would expect it to only pickup data to the vpn server without modifications... or the network_interface may be different in your setup.

[TheLostLambda]

@jconley

Hmm, I'll be honest, I'm not sure I know what you mean. In my case, the VPN server is also the sftp server. I'll drop some info here that might be helpful? If not, just let me know what you need :)

My interface:

> ip a
...
8: wg0: <POINTOPOINT,NOARP,UP,LOWER_UP> mtu 1420 qdisc noqueue state UNKNOWN group default qlen 1000
    link/none 
    inet 10.0.0.2/32 scope global noprefixroute wg0
       valid_lft forever preferred_lft forever
...

My test file:

> dd if=/dev/random of=100MB-Test bs=1M count=100
> ls 100MB-Test -lah
-rw-r--r-- 1 tll tll 100M Sep 29 18:16 100MB-Test

SCP command:

scp 100MB-Test 10.0.0.1:/tmp/

Bandwhich lines:

ssh     99.22MiB / 1.30MiB

10.0.0.1     100.47MiB / 1.19MiB

<wg0>:57716 => 10.0.0.1:22 (tcp)     ssh     100.47MiB / 1.19MiB

There is additional, non-ssh traffic picked up there, but I'm not certain that's what you were looking for. Let me know if you'd like some different information.

[thepacketgeek]

Thanks @TheLostLambda and @jconley for clarification of Wireguard only, I think I meant to say (happens with wireguard but not all utun interfaces globally).

I have a draft PR (#192) adding this flag (for target_os = "macos" only), is this the right direction for handling the optional payload offset?

[TheLostLambda]

@jconley Oh dear, I'm realising it's possible I've crashed into a MacOS debugging thread with Linux. I'm sorry, I probably should have taken a closer look at the issue 😅

All of my testing here has been on Linux, so I'm sorry if that's sparked confusion!

[imsnif]

Hey, so - before I move to reviewing the PR. So that I understand, what's the situation now? What sort of interfaces/os combinations need the offset and which don't?

[thepacketgeek]

@imsnif I think we're still trying to figure that out. @jconley brought up the point that libpnet only seems to add offsets for loopbacks. This lines up with both of our testing of #192 (utun sniffing only works with --no-payload-offset flag), and is possibly a result of the change from libpnet 0.23 -> 0.26.

@zhangxp1998 if you have a chance to test #192, can you confirm if your utun interfaces only work with the flag (indicating offset is only required for loopbacks)?