Fix bug that non-connected Udp sockets aren't displayed #82

zhangxp1998 · 2020-01-06T23:01:26Z

This is in response to #81

With this change bandwhich will correctly display UDP traffic sent from sockets that are not connected to any remote port.

imsnif · 2020-01-07T07:24:38Z

Wow, thanks for all the quick work on this!

This is a big change in mostly untested code. I would like to go over this before it's merged. If @ebroto wants to as well (and has the time) that would of course be great :)

ebroto · 2020-01-07T07:29:25Z

Will do this evening if that's OK :)

imsnif · 2020-01-07T07:34:02Z

Will do this evening if that's OK :)

That's my plan as well after taking care of #51 one way or the other.

imsnif

Hey, this looks great. I think you've done good work on both parts of this (the lsof issue and the data structure change).

I like the new approach of keeping track of process names. I think it's much more robust and don't feel it's prone to misidentification. iirc from my networking days, I think a bound local port is unique for an interface - as you mentioned in the other thread.

I also tested this out on my machine and it behaves well. I see some UNKNOWNs here and there, but I think we can squash that behaviour later if it gets in the way (or solve the bugs if we find we're mislabeling them).

I left some comments, otherwise LGTM

imsnif · 2020-01-07T18:36:19Z

src/display/ui_state.rs

+        if let Some(process_name) = connections_to_procs.get(local_socket) {
+            Some(process_name)
+        } else if let Some(process_name) = connections_to_procs.get(&LocalSocket {
+            ip: IpAddr::V4(Ipv4Addr::new(0, 0, 0, 0)),


If I understand correctly, we always have the info of whether a LocalSocket is v4 or v6, so can we make an "IpVersion" enum in LocalSocket, then we won't need these conditionals?

Technically, we don't need the extra enum. Because IpAddr itself knows whether it is v4 or v6. We can figure out whether a local socket is v4 or v6, but still, we must conditionally create a v4 wildcard(0.0.0.0) or a v6 wildcard(::0).

src/main.rs

imsnif · 2020-01-07T18:49:12Z

src/network/connection.rs

 pub enum Protocol {
    Tcp,
    Udp,
+    Icmp,


thumbs up

imsnif · 2020-01-07T18:54:05Z

src/network/utilization.rs

@@ -2,14 +2,14 @@ use crate::network::{Connection, Direction, Segment};

 use ::std::collections::HashMap;

-#[derive(Clone)]
+#[derive(Clone, Debug)]


Can we remove these?

I think cloned is used in clone_and_reset + I would say Debug never hurts

I was referring to the debug... hmm - doesn't it add needless stuff? (I honestly don't know - asking :) )

It adds an impl Debug for X which is handy to use with dbg! among other use cases. The thing is that if you don't impl it for a struct member, you can naturally not have it automatically impl'd for the struct. Maybe personal taste though.

Aha - and so it doesn't add anything to the binary unless there's any use of it (with the macro of {:?} for example)? So... why don't we derive it for everything?

Not sure if it would get optimized out TBH.

Well, there's some influential people that thinks we should, see e.g. Mr. BurntSushi

imsnif · 2020-01-07T18:58:02Z

src/os/lsof_utils.rs

+        // "(LISTEN)" or "(ESTABLISHED)",  this column may or may not be present
+        // let connection_state = columns[9];
+        // If this socket is in a "connected" state
+        if let Some(caps) = CONNECTION_REGEX.captures(connection_str) {


The comments above are great. Can we add another one here with a sample row matching this regex?

imsnif · 2020-01-07T19:01:59Z

src/os/lsof_utils.rs

        }
    }

    pub fn get_protocol(&self) -> Protocol {
        return Protocol::from_str(&self.protocol).unwrap();
    }

-    pub fn get_ip_address(&self) -> IpAddr {
-        return IpAddr::V4(self.ip.parse().unwrap());
+    pub fn get_remote_ip(&self) -> IpAddr {


Are these a performance optimization?

No, they are just an attempt to parse both Ipv4 and ipv6 addresses.

src/os/macos.rs

imsnif · 2020-01-07T19:29:21Z

src/tests/cases/raw_mode.rs

@@ -105,7 +105,7 @@ fn multiple_packets_of_traffic_from_different_connections() {
            "2.2.2.2",
            "10.0.0.2",
            54321,
-            443,
+            4434,


I'm not sure I understand the changes in the tests - could you explain them a little?

In the old test cases, multiple processes are owning local port 443, but each process is connected to a different remote socket. This simulated scenario does not go well with the new logic of identifying processes, since we now expect a local port to be owned by a single process. So I changed the test cases so that each process is on a different port.

we now expect a local port to be owned by a single process.

I'm thinking out loud, correct me if I'm wrong, but can't you assign multiple IPs to a single network interface? In that case, can't two processes use the same local port for the different IPs?

but can't you assign multiple IPs to a single network interface

To the best of my knowledge, you can't. Different interfaces(WiFi/ethernet) can have different IPs. Which means 1 process can use port 443 of WiFi, and another one can use port 443 of ethernet. When identifying processes, local_ip is used along with port number and protocol, so we should be able to handle this situation correctly.

The ip command allows you to add multiple IPs to the same interface, but digging a bit into it I think it would create an "alias", so maybe the interface name would be different for our purposes, see here for example.

In any case, I think that's a niche case, not going to nitpick :) But I will try to setup this just to test how it works if I have time.

Hmm can you make 1 interface have more than 1 external IPs?(how does routing/ISP work then?)

I also know that this is possible (eg. on routers - they cannot be limited by their physical interfaces, otherwise it would make things very difficult) - but I have a feeling @Alcaro is more in on the nitty-gritty details here than I am.

But indeed, very niche for our use-case.

Routers? IIRC NAT boxes themselves typically have 1 IP? But they modify the packet's destination/source when forwarding packets. Not sure why non-NAT box routers need to deal with multiple IP on same interface though

One example: https://learningnetwork.cisco.com/thread/21286

Hmm can you make 1 interface have more than 1 external IPs?

Does IPv4 plus IPv6 count?

If no, still possible, but a lot less common.

sudo ip addr add 192.168.3.1/28 dev enp1s0f0
sudo ip addr add 192.168.4.1/28 dev enp1s0f0

Pingable and bindable, and most likely externally reachable if I had a suitably configurable router. Can't get IPv6 working that way, though - they show up in ifconfig, but I get EADDRNOTAVAIL if I try to bind them. No idea why, networking is tricky.

(how does routing/ISP work then?)

Routing/forwarding mostly stays in the kernel, it doesn't show up in lsof. Copying the packets to userspace would be a performance sinkhole. (Maintaining the NAT table may be partially in userspace, not sure.)

Hmm can you make 1 interface have more than 1 external IPs?

Does IPv4 plus IPv6 count?

If no, still possible, but a lot less common.

sudo ip addr add 192.168.3.1/28 dev enp1s0f0
sudo ip addr add 192.168.4.1/28 dev enp1s0f0

Pingable and bindable, and most likely externally reachable if I had a suitably configurable router. Can't get IPv6 working that way, though - they show up in ifconfig, but I get EADDRNOTAVAIL if I try to bind them. No idea why, networking is tricky.

(how does routing/ISP work then?)

Routing/forwarding mostly stays in the kernel, it doesn't show up in lsof. Copying the packets to userspace would be a performance sinkhole. (Maintaining the NAT table may be partially in userspace, not sure.)

sudo ip addr add 192.168.3.1/28 dev enp1s0f0 sudo ip addr add 192.168.4.1/28 dev enp1s0f0

I suppose this only adds an additional IP behind the NAT box(AKA private IP)?

Of course it's not globally accessible. If it was, I'd add 8.8.8.8 to my machine and see how much of the internet breaks.

I'm not even sure if it'd work in the LAN. I think it'd work on most routers, but probably not all - it depends on how they handle ARP requests. (The chosen addresses must be within the router's netmask, otherwise it'll send the packets in wrong direction.)

I can't test locally - the extra addresses I add don't stick. Either something (NetworkManager?) is instantly undoing my config, or ip addr silently fails. (My previous tests were done on an unplugged ethernet port.)

Of course such a configuration is extremely rare outside experimental conditions, even without unrelated processes sharing a port number. If supporting it would be troublesome, there's no need to.

ebroto

Great work! Overall LGTM. It seems to work properly on Linux, for TCP and UDP. If I'm not mistaken there is no support in procfs::net for ICMP.

I left some nits :)

src/os/lsof_utils.rs

src/network/connection.rs

src/os/lsof_utils.rs

src/display/ui_state.rs

imsnif

When you feel this is ready to be merged - go for it. I'm satisfied once the CI passes.

imsnif · 2020-01-07T21:25:09Z

Great work, btw! Be sure to add it to the changelog.

zhangxp1998 requested a review from ebroto January 6, 2020 23:01

zhangxp1998 force-pushed the udp branch 8 times, most recently from 2f7807b to fd5bacf Compare January 7, 2020 01:43

zhangxp1998 mentioned this pull request Jan 7, 2020

Display weighted average of bandwidth #77

Merged

imsnif self-requested a review January 7, 2020 07:23

imsnif requested changes Jan 7, 2020

View reviewed changes

ebroto requested changes Jan 7, 2020

View reviewed changes

zhangxp1998 force-pushed the udp branch 3 times, most recently from 9303480 to a445b0e Compare January 7, 2020 20:29

zhangxp1998 added 2 commits January 7, 2020 15:31

Add local_ip field to Connection/RawConnection struct

fc39cff

Update parsing code to parse local_ip

66c0347

zhangxp1998 force-pushed the udp branch 6 times, most recently from 78150a6 to 26ad0c0 Compare January 7, 2020 21:08

ebroto self-requested a review January 7, 2020 21:20

ebroto approved these changes Jan 7, 2020

View reviewed changes

imsnif approved these changes Jan 7, 2020

View reviewed changes

zhangxp1998 added 3 commits January 7, 2020 16:27

Use <local_ip, local_port, protocol> to indentify processes

2a30061

Update snapshots because new process identification logic changes output

9737ba9

Format code using rustfmt

cf772ac

zhangxp1998 force-pushed the udp branch from 26ad0c0 to 175fae4 Compare January 7, 2020 21:28

Address some comments on PR

471fd86

zhangxp1998 force-pushed the udp branch from 175fae4 to 471fd86 Compare January 7, 2020 21:40

zhangxp1998 merged commit 5826f04 into imsnif:master Jan 7, 2020

zhangxp1998 mentioned this pull request Jan 7, 2020

Failing to report unbound UDP traffic #81

Closed

zhangxp1998 deleted the udp branch January 9, 2020 23:50

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix bug that non-connected Udp sockets aren't displayed #82

Fix bug that non-connected Udp sockets aren't displayed #82

zhangxp1998 commented Jan 6, 2020

imsnif commented Jan 7, 2020

ebroto commented Jan 7, 2020

imsnif commented Jan 7, 2020

imsnif left a comment

imsnif Jan 7, 2020

zhangxp1998 Jan 7, 2020

imsnif Jan 7, 2020

imsnif Jan 7, 2020

zhangxp1998 Jan 7, 2020

ebroto Jan 7, 2020

imsnif Jan 7, 2020

ebroto Jan 7, 2020

imsnif Jan 7, 2020

ebroto Jan 7, 2020

imsnif Jan 7, 2020

zhangxp1998 Jan 7, 2020

imsnif Jan 7, 2020

zhangxp1998 Jan 7, 2020

imsnif Jan 7, 2020

zhangxp1998 Jan 7, 2020

ebroto Jan 7, 2020 •

edited

Loading

zhangxp1998 Jan 7, 2020

ebroto Jan 7, 2020

zhangxp1998 Jan 7, 2020

imsnif Jan 7, 2020

Alcaro Jan 7, 2020

zhangxp1998 Jan 7, 2020

Alcaro Jan 8, 2020

ebroto left a comment

imsnif left a comment

imsnif commented Jan 7, 2020

Fix bug that non-connected Udp sockets aren't displayed #82

Fix bug that non-connected Udp sockets aren't displayed #82

Conversation

zhangxp1998 commented Jan 6, 2020

imsnif commented Jan 7, 2020

ebroto commented Jan 7, 2020

imsnif commented Jan 7, 2020

imsnif left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebroto Jan 7, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ebroto left a comment

Choose a reason for hiding this comment

imsnif left a comment

Choose a reason for hiding this comment

imsnif commented Jan 7, 2020

ebroto Jan 7, 2020 •

edited

Loading