Since ntopng 3.2.171211 and nprobe 8.2.171211 stable, no flows (NETFLOW)

[harmane]

Sorry to ask this here but since upgrading from 3.1.x & 8.1.x to 3.2.171211 & 8.2.171211 respectively I cannot get netflow data to be available to ntopng from nprobe. I have ENT license for ntopng and pro for nprobe:

ntopng --version v.3.2.171214 [Enterprise/Professional build]
nprobe --version v.8.2.171214 (r5982)

ntopng.conf

-d=/ntopng-data/ntopng
-G=/ntopng-data/ntopng.pid
-i=tcp://127.0.0.1:5556
-S=local
#--dns-mode=3
-m=LOCAL&PUB_NETWORKS
-X=250000
-x=250000

nprobe-none.conf

--zmq="tcp://127.0.0.1:5556"
--collector-port=2055
-n=none
-i=none
-V=9
-T="%IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV4_SRC_MASK %IPV4_DST_MASK %L4_SRC_PORT %L4_DST_PORT %IPV6_SRC_ADDR %IPV6_DST_ADDR %IPV6_SRC_MASK %IPV6_DST_MASK %IP_PROTOCOL_VERSION %SRC_TOS %PROTOCOL %ICMP_TYPE %INPUT_SNMP %SRC_AS %DST_AS %IPV4_NEXT_HOP %IPV6_NEXT_HOP %TCP_FLAGS %OUTPUT_SNMP %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %MIN_TTL %MAX_TTL %FIRST_SWITCHED %LAST_SWITCHED %SRC_VLAN %DST_VLAN %DOT1Q_SRC_VLAN %DOT1Q_DST_VLAN %EXPORTER_IPV4_ADDRESS %IN_SRC_MAC %IN_DST_MAC"

I see this in nprobe --help which seems interesting:

--zmq-probe-mode
By default nProbe in ZMQ mode acts as a server with subscribers
(e.g. ntopng) attaching to it. When this option is used, roles are
reverted (i.e. use ntopng --zmq-collector-mode).

I also see the interface created in ntopng configs on various issue postings here presented as tcp://127.0.0.1:5556c which I believe is also illustrated in a commit in 2016 deprecating --zmq-collector-mode. Can't find link currently that points to this. Unfortunately for me ntopng fails to start when trying to create that interface.

I've tried starting nprobe via cli:

nprobe -i none -n none -V 9 -T=%IPV4_SRC_ADDR %IPV4_DST_ADDR %IPV4_SRC_MASK %IPV4_DST_MASK %L4_SRC_PORT %L4_DST_PORT %IPV6_SRC_ADDR %IPV6_DST_ADDR %IPV6_SRC_MASK %IPV6_DST_MASK %IP_PROTOCOL_VERSION %SRC_TOS %PROTOCOL %ICMP_TYPE %INPUT_SNMP %SRC_AS %DST_AS %IPV4_NEXT_HOP %IPV6_NEXT_HOP %TCP_FLAGS %OUTPUT_SNMP %IN_BYTES %IN_PKTS %OUT_BYTES %OUT_PKTS %MIN_TTL %MAX_TTL %FIRST_SWITCHED %LAST_SWITCHED %SRC_VLAN %DST_VLAN %DOT1Q_SRC_VLAN %DOT1Q_DST_VLAN %EXPORTER_IPV4_ADDRESS %IN_SRC_MAC %IN_DST_MAC" --zmq "tcp://127.0.0.1:5556 --zmq-probe-mode

...and still no joy.

tcpdump port 2055 on host sure enough shows huge stream of data coming from my core switch to ntopng VM.

[simonemainardi]

add option --zmq-probe-mode to nProbe and append a c (which stands for collector) to the ntopng zmq endpoint. Assuming both apps are on the same machine, you can do a

./nprobe --zmq-probe-mode --zmq tcp://127.0.0.1:5556 .....
./ntopng -i tcp://*:5556c .....

also it looks like there's a double quote unclosed in your nprobe configuration ... --zmq "tcp://127.0.0.1:5556 --zmq-probe-mode ...

if you are still not getting the flows properly visualized, check with tcpdump on port 5556 and see if there are packets there.

[harmane]

Interesting. The unclosed quotes were a typo.

On my test box (community) I initially had an improperly setup symlink in multi-user.target.wants for nprobe-none.conf. Once fixed I would get segfault when running ntopng until I cleared application data. I could then see flows in ntopng.

On the enterprise instance I confirmed everything from test and cleared application data but still received same segfault until I added -i=tcp://*:5556c in ntopng.conf and --zmq-probe-mode in nprobe-none.conf.

Everything seems to be working now, thank you.

[harmane]

Spoke too soon. Segfault after a few minutes:

Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffb6ff5700 (LWP 2697)]
__memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:157
157 ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S: No such file or directory.
(gdb) bt
#0 __memcpy_sse2_unaligned () at ../sysdeps/x86_64/multiarch/memcpy-sse2-unaligned.S:157
#1 0x00000000004d0937 in FlowProfiles::getFlowProfile (this=0x1db62a0, flow=0x7fff78154620) at pro/FlowProfiles.cpp:48
#2 0x000000000043900c in NetworkInterface::getFlowProfile (this=0x823670, f=0x7fff78154620) at /home/deri/ntopng/include/NetworkInterface.h:536
#3 0x0000000000439b69 in Flow::updateProfile (this=0x7fff78154620) at /home/deri/ntopng/include/Flow.h:398
#4 0x000000000042e964 in Flow::setDetectedProtocol (this=0x7fff78154620, proto_id=..., forceDetection=true) at src/Flow.cpp:575
#5 0x000000000042d2b1 in Flow::Flow (this=0x7fff78154620, _iface=0x823670, _vlanId=0, _protocol=2 '\002', _cli_mac=0x7fff877ed260, _cli_ip=0x7fffb6ff2650, _cli_port=0, _srv_mac=0x7fffa4012990, _srv_ip=0x7fffb6ff2630, _srv_port=37888, _first_seen=1513602968,
_last_seen=1513602998) at src/Flow.cpp:136
#6 0x00000000004477b1 in NetworkInterface::getFlow (this=0x823670, srcMac=0x7fff877ed260, dstMac=0x7fffa4012990, vlan_id=0, deviceIP=176750593, inIndex=69, outIndex=0, src_ip=0x7fffb6ff2650, dst_ip=0x7fffb6ff2630, src_port=0, dst_port=37888, l4_proto=2 '\002',
src2dst_direction=0x7fffb6ff267e, first_seen=1513602968, last_seen=1513602998, rawsize=0, new_flow=0x7fffb6ff267d) at src/NetworkInterface.cpp:894
#7 0x00000000004490bc in NetworkInterface::processFlow (this=0x823670, zflow=0x7fffb6ff2720) at src/NetworkInterface.cpp:1140
#8 0x00000000004a2d02 in ParserInterface::parseSingleFlow (this=0x823670, o=0x7fff86c80480, source_id=0 '\000', iface=0x823670) at src/ParserInterface.cpp:939
#9 0x00000000004a2e43 in ParserInterface::parseFlow (this=0x823670,
payload=0x7fff874e3ef0 "[{"8":"10.136.86.26","12":"239.255.255.250","9":23,"13":0,"7":0,"11":148,"27":"::","28":"::","29":0,"30":0,"60":4,"5":0,"4":2,"32":0,"10":389,"16":0,"17":0,"15":"0.0.0.0","6":16,"14":0,"1":32,"2":1,"2"..., payload_size=3788, source_id=0 '\000', data=0x823670) at src/ParserInterface.cpp:972
#10 0x00000000004be84c in CollectorInterface::collect_flows (this=0x823670) at src/CollectorInterface.cpp:229
#11 0x00000000004be983 in packetPollLoop (ptr=0x823670) at src/CollectorInterface.cpp:260
#12 0x00007ffff6a5b064 in start_thread (arg=0x7fffb6ff5700) at pthread_create.c:309
#13 0x00007ffff41b562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

[emanuele-f]

Can you provide a pcap file with sample traffic to reproduce the crash?

[harmane]

Sent via email w/pcap attached.

[emanuele-f]

Thank you, the issue as been addressed with commits 188c8c4 and c6c74eb . A new package will be available in one hour.

[harmane]

@emanuele-f

Sorry to keep replying to this. I have updated to lastest versions from today and ntopng started but after reboot or stop/start I get segfault:

19/Dec/2017 10:10:52 [NetworkInterface.cpp:2326] Started packet polling on interface tcp://:5556c [id: 5]...
19/Dec/2017 10:10:52 [CollectorInterface.cpp:122] Collecting flows on tcp://:5556c
Program received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fffb6ff5700 (LWP 1428)]
0x000000000042c5e8 in Mac::getDeviceType (this=0x0) at /home/deri/ntopng/include/Mac.h:103
103 /home/deri/ntopng/include/Mac.h: No such file or directory.
(gdb) bt
#0 0x000000000042c5e8 in Mac::getDeviceType (this=0x0) at /home/deri/ntopng/include/Mac.h:103
#1 0x000000000043790f in Flow::updateFlowShapers (this=0x7fffa4013310) at src/Flow.cpp:2838
#2 0x000000000042e93c in Flow::setDetectedProtocol (this=0x7fffa4013310, proto_id=..., forceDetection=true) at src/Flow.cpp:564
#3 0x00000000004497d3 in NetworkInterface::processFlow (this=0x8236b0, zflow=0x7fffb6ff2710) at src/NetworkInterface.cpp:1279
#4 0x00000000004a303b in ParserInterface::parseSingleFlow (this=0x8236b0, o=0x7fffa4001fe0, source_id=0 '\000', iface=0x8236b0) at src/ParserInterface.cpp:967
#5 0x00000000004a3183 in ParserInterface::parseFlow (this=0x8236b0,
payload=0x7fffa4000a20 "[{"8":"17.248.135.74","12":"10.135.145.55","9":0,"13":20,"7":443,"11":57218,"27":"::","28":"::","29":0,"30":0,"60":4,"5":0,"4":6,"32":0,"10":317,"16":714,"17":0,"15":"10.135.145.55","6":0,"14":66,"1":"..., payload_size=3902, source_id=0 '\000', data=0x8236b0) at src/ParserInterface.cpp:1001
#6 0x00000000004bebcc in CollectorInterface::collect_flows (this=0x8236b0) at src/CollectorInterface.cpp:229
#7 0x00000000004bed03 in packetPollLoop (ptr=0x8236b0) at src/CollectorInterface.cpp:260
#8 0x00007ffff6a5b064 in start_thread (arg=0x7fffb6ff5700) at pthread_create.c:309
#9 0x00007ffff41b562d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111

This requires removing /var/lib/redis/dump.rdb which clears all my settings. ntopng will start however after doing so. I did not need to remove application data in ntopng dir.

[emanuele-f]

If you manage to reproduce the crash again please see if it still happens after you remove the -S option

[emanuele-f]

Possibly related to #1610

[emanuele-f]

@harmane the above issue is fixed with commit df7a7c6 . Please update to the new version, which will be available in one hour.