Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

JSON Format Unavailable with Aggregation Option (-A) #575

Closed
remontti opened this issue Nov 2, 2024 · 10 comments
Closed

JSON Format Unavailable with Aggregation Option (-A) #575

remontti opened this issue Nov 2, 2024 · 10 comments

Comments

@remontti
Copy link

remontti commented Nov 2, 2024

Hello,

After updating to version 1.7.5 of nfdump, I noticed that it’s no longer possible to generate JSON output when using the aggregation option -A.

Command used:

nfdump -r 'nfcapd.202411021440' -n 5 -O bytes -A srcas 'any' -o json

Return:

Can not use print format json to aggregate flows

It works fine with -o csv, but it would be helpful to have JSON output available with -A as well. Is this behavior expected in version 1.7.5? Are there any plans to reintroduce JSON output with the aggregation option -A?

Thank you for the excellent work on nfdump!
@remontti
Copy link
Author

remontti commented Nov 5, 2024

I will also take this opportunity to ask

Analyze traffic with the following query:

nfdump -M '/var/rr-flows/RS-BORDA' -R '2024/11/04/nfcapd.202411041959:2024/11/04/nfcapd.202411042359' -n 3 -6 -A srcip -O bytes 'src ip 8.8.8.8 and proto icmp'

Comparison of the outputs:

Version 1.7.4

Date first seen         Duration                                      Src IP Addr   Packets    Bytes      bps    Bpp Flows
2024-11-04 19:59:50.000     03:59:20.000                                  8.8.8.8    724992   59.0 M    32858     81   701
Summary: total flows: 701, total bytes: 59.0 M, total packets: 724992, avg bps: 32858, avg pps: 50, avg bpp: 81

Version 1.7.5

Date first seen         Duration                                      Src IP Addr   Packets    Bytes      bps    Bpp Flows
2024-11-04 19:59:50.000     03:59:20.000                                  8.8.8.8    724992   59.0 M    32858     81     1
Summary: total flows: 701, total bytes: 59.0 M, total packets: 724992, avg bps: 32858, avg pps: 50, avg bpp: 81

As you can see, while the total flows count remains correct in the summary (total flows: 701), the value in the Flows field for the IP 8.8.8.8 has changed from 701 to 1.

Questions:

  1. Is this change in the detailed Flows count intentional in this version?
  2. Has there been any modification to the grouping or display criteria that would explain this difference?

Thank you for your attention!

@phaag
Copy link
Owner

phaag commented Nov 6, 2024

The output format json is currently not supported for aggregated flows. One goal was to speed up json for full records. It needs a bit more works to fix that.

As of the differences, I will check that and will be back to you as I am out of office.

@phaag
Copy link
Owner

phaag commented Nov 9, 2024

As of the flow count - I cannot reproduce this - I always get the correct flow count. I you can create a reproducible scenario with a file you could send me, this would be appreciated.

@remontti
Copy link
Author

remontti commented Nov 9, 2024

I set up 2 servers to receive the flows; the router sending them is a HUAWEI NetEngine 8000 F1A-8H20Q.

Huawei Versatile Routing Platform Software
VRP (R) software, Version 8.200 (NetEngine 8000 V800R012C10SPC300)
Copyright (C) 2012-2020 Huawei Technologies Co., Ltd.
HUAWEI NetEngine 8000 F1A-8H20Q uptime is 702 days, 22 hours, 0 minute
Patch Version: V800R012SPH058

SERVER 1️⃣ (1.7.5)

root@flow-upgrade# nfdump -V
nfdump: Version: 1.7.5-release Options: NSEL-NEL Date: Wed Oct 23 19:53:03 CEST 2024
COLLECTOR
/usr/local/bin/nfcapd -6 -D -p 3333 -B 67108864 -t 60 -s 1024 -S 1 -P /var/rr-flows/RS-BORDA/capd_3333.pid -z=lz4 -I RS-BORDA -w /var/rr-flows/RS-BORDA
root@flow-upgrade[/var/rr-flows/RS-BORDA/2024/11/09]# l -tr nfcapd.20241109153*
-rw-r--r-- 1 root root 911K nov  9 15:31 nfcapd.202411091530
-rw-r--r-- 1 root root 897K nov  9 15:32 nfcapd.202411091531
-rw-r--r-- 1 root root 911K nov  9 15:33 nfcapd.202411091532
-rw-r--r-- 1 root root 888K nov  9 15:34 nfcapd.202411091533
-rw-r--r-- 1 root root 914K nov  9 15:35 nfcapd.202411091534
-rw-r--r-- 1 root root 878K nov  9 15:36 nfcapd.202411091535
-rw-r--r-- 1 root root 897K nov  9 15:37 nfcapd.202411091536
-rw-r--r-- 1 root root 916K nov  9 15:38 nfcapd.202411091537
-rw-r--r-- 1 root root 877K nov  9 15:39 nfcapd.202411091538
-rw-r--r-- 1 root root 884K nov  9 15:40 nfcapd.202411091539
root@flow-upgrade[/var/rr-flows/RS-BORDA/2024/11/09]# nfdump -r nfcapd.202411091530  -O flows -A srcip -n 10
Date first seen         Duration               Src IP Addr   Packets    Bytes      bps    Bpp Flows
2024-11-09 15:28:45.000     00:01:48.000     177.75.149.17     2.9 M    3.7 G  273.8 M   1272     1
2024-11-09 15:28:46.000     00:01:56.000     177.75.149.32     2.1 M    2.6 G  180.4 M   1255     1
2024-11-09 15:28:47.000     00:01:44.000    179.127.141.36     1.6 M    1.9 G  149.3 M   1248     1
2024-11-09 15:28:48.000     00:01:49.000    179.127.136.40     2.0 M    2.7 G  196.6 M   1340     1
2024-11-09 15:28:45.000     00:01:59.000    179.127.136.41     2.0 M    2.7 G  182.0 M   1323     1
2024-11-09 15:28:53.000     00:01:42.000    179.127.141.17     1.6 M    1.9 G  151.8 M   1244     1
2024-11-09 15:28:46.000     00:01:43.000   179.127.128.143     1.4 M    1.8 G  143.4 M   1316     1
2024-11-09 15:28:46.000     00:01:55.000   179.127.128.142     1.0 M    1.4 G   94.3 M   1332     1
2024-11-09 15:28:54.000     00:01:39.000   179.127.128.140     1.1 M    1.4 G  113.8 M   1335     1
2024-11-09 15:28:57.000     00:01:38.000     71.18.118.229    633856  852.3 M   69.6 M   1344     1
Summary: total flows: 22348, total bytes: 64.5 G, total packets: 65.9 M, avg bps: 4.2 G, avg pps: 535618, avg bpp: 979
Time window: 2024-11-09 15:28:41 - 2024-11-09 15:30:44, Duration:    00:02:03.000
Total records processed: 22348, passed: 22348, Blocks skipped: 0, Bytes read: 4344600
Sys: 0.0042s User: 0.0166s Wall: 0.0136s flows/second: 1643827.0 Runtime: 0.014
roo@flow-upgrade[~]# nfcapd -E -p 3333 -w /tmp/
Verbose log level: 3
Add flow source: ident: none, IP: any IP, flowdir: /tmp
Bound to IPv4 host/IP: any, Port: 3333
Init v1
Init v5/v7: Default sampling: 1
Init v9: Max number of v9 tags enabled: 114, default sampling: 1
Init IPFIX: Max number of ipfix tags enabled: 113, default sampling: 1
Startup nfcapd.
Process_v9: New v9 exporter: SysID: 1, Domain: 18841857, IP: x.x.x.255
Process_v9: New v9 exporter: SysID: 2, Domain: 2166325505, IP: x.x.x.255

Flow Record: 
  Flags        =              0x00 NETFLOW v9, Unsampled
  Elements     =                10: 1 3 4 5 6 7 9 11 12 42 
  size         =               228
  engine type  =               129
  engine ID    =                 1
  export sysid =                 2
  first        =     1731178415000 [2024-11-09 15:53:35.000]
  last         =     1731178415000 [2024-11-09 15:53:35.000]
  received at  =     1731178434554 [2024-11-09 15:53:54.554]
  proto        =                 6 TCP
  tcp flags    =              0x11 ...A...F
  src port     =               443
  dst port     =             60048
  src tos      =                 0
  in packets   =                 1
  in bytes     =                90
  src addr     =  2600:1901:0:aab1::
  dst addr     =  2804:xxxx:8003:400:a484:316a:95ab:c553
  input        =               153
  output       =               167
  src mask     =                32 0:aab1::/32
  dst mask     =                56 8003:400::/56
  fwd status   =               144
  dst tos      =               234
  direction    =                 0
  biFlow Dir   =              0x00
  end reason   =              0x00
  out packets  =                 0
  out bytes    =                 0
  aggr flows   =                 1
  src vlan     =               413
  dst vlan     =                 0
  src as       =             396982
  dst as       =             65530
  bgp next hop =  fd00:bacb:50:24::2
  ip next hop  =  fd00:bacb:50:24::2
  ip exporter  =   x.x.x.255

Flow Record: 
  Flags        =              0x00 NETFLOW v9, Unsampled
  Elements     =                10: 1 3 4 5 6 7 9 11 12 42 
  size         =               228
  engine type  =               129
  engine ID    =                 1
  export sysid =                 2
  first        =     1731178415000 [2024-11-09 15:53:35.000]
  last         =     1731178415000 [2024-11-09 15:53:35.000]
  received at  =     1731178434554 [2024-11-09 15:53:54.554]
  proto        =                 6 TCP
  tcp flags    =              0x11 ...A...F
  src port     =             60298
  dst port     =               443
  src tos      =                 0
  in packets   =                 1
  in bytes     =                90
  src addr     =  2804:xxxx:acca:1900:4b:179d:9fb2:e842
  dst addr     =  2a03:2880:f248:1c9:face:b00c:0:43fe
  input        =               142
  output       =               153
  src mask     =                40 acca:1900::/40
  dst mask     =                48 f248:1c9:4b::/48
  fwd status   =               187
  dst tos      =                 1
  direction    =                 1
  biFlow Dir   =              0x00
  end reason   =              0x00
  out packets  =                 0
  out bytes    =                 0
  aggr flows   =                 1
  src vlan     =               447
  dst vlan     =                 0
  src as       =             65530
  dst as       =             32934
  bgp next hop =  fd00:bacb:50:16::2
  ip next hop  =  fd00:bacb:50:16::2
  ip exporter  =   x.x.x.255

SERVER 2️⃣ (1.7.4)

root@flow# nfdump -V
nfdump: Version: 1.7.4-release Options: NSEL-NEL Date: Sat Feb 17 15:05:20 CET 2024
COLLECTOR
/usr/local/bin/nfcapd -6 -D -p 3055 -B 67108864 -t 60 -s 1024 -S 1 -P /var/rr-flows/RS-BORDA/capd_3055.pid -z=lz4 -I RS-BORDA -w /var/rr-flows/RS-BORDA
root@flow[/var/rr-flows/RS-BORDA/2024/11/09]# l -tr nfcapd.20241109153*
-rw-r--r-- 1 root root 840K nov  9 15:31 nfcapd.202411091530
-rw-r--r-- 1 root root 825K nov  9 15:32 nfcapd.202411091531
-rw-r--r-- 1 root root 838K nov  9 15:33 nfcapd.202411091532
-rw-r--r-- 1 root root 816K nov  9 15:34 nfcapd.202411091533
-rw-r--r-- 1 root root 841K nov  9 15:35 nfcapd.202411091534
-rw-r--r-- 1 root root 809K nov  9 15:36 nfcapd.202411091535
-rw-r--r-- 1 root root 826K nov  9 15:37 nfcapd.202411091536
-rw-r--r-- 1 root root 844K nov  9 15:38 nfcapd.202411091537
-rw-r--r-- 1 root root 810K nov  9 15:39 nfcapd.202411091538
-rw-r--r-- 1 root root 810K nov  9 15:40 nfcapd.202411091539
root@flow[/var/rr-flows/RS-BORDA/2024/11/09]# nfdump -r nfcapd.202411091530  -O flows -A srcip -n 10 
Date first seen         Duration               Src IP Addr   Packets    Bytes      bps    Bpp Flows
2024-11-09 15:28:45.000     00:01:48.000     177.75.149.17     2.9 M    3.7 G  273.8 M   1272   589
2024-11-09 15:28:46.000     00:01:56.000     177.75.149.32     2.1 M    2.6 G  180.4 M   1255   524
2024-11-09 15:28:47.000     00:01:44.000    179.127.141.36     1.6 M    1.9 G  149.3 M   1248   425
2024-11-09 15:28:48.000     00:01:49.000    179.127.136.40     2.0 M    2.7 G  196.6 M   1340   413
2024-11-09 15:28:45.000     00:01:59.000    179.127.136.41     2.0 M    2.7 G  182.0 M   1323   411
2024-11-09 15:28:53.000     00:01:42.000    179.127.141.17     1.6 M    1.9 G  151.8 M   1244   400
2024-11-09 15:28:46.000     00:01:43.000   179.127.128.143     1.4 M    1.8 G  143.4 M   1316   244
2024-11-09 15:28:46.000     00:01:55.000   179.127.128.142     1.0 M    1.4 G   94.3 M   1332   218
2024-11-09 15:28:54.000     00:01:39.000   179.127.128.140     1.1 M    1.4 G  113.8 M   1335   199
2024-11-09 15:28:57.000     00:01:38.000     71.18.118.229    633856  852.3 M   69.6 M   1344   185
Summary: total flows: 22348, total bytes: 64.5 G, total packets: 65.9 M, avg bps: 4.2 G, avg pps: 535618, avg bpp: 979
Time window: 2024-11-09 15:28:41 - 2024-11-09 15:30:44
Total flows processed: 22348, passed: 22348, Blocks skipped: 0, Bytes read: 4165816
Sys: 0.0032s User: 0.0097s Wall: 0.0094s flows/second: 2377414.1 Runtime: 0.0099s
root@flow# nfcapd -E -p 3055 -w /tmp/
Verbose log level: 3
Add flow source: ident: none, IP: any IP, flowdir: /tmp
Bound to IPv4 host/IP: any, Port: 3055
Init v1
Init v5/v7: Default sampling: 1
Init v9: Max number of v9 tags enabled: 106, default sampling: 1
Init IPFIX: Max number of ipfix tags enabled: 97, default sampling: 1
Startup nfcapd.
Process_v9: New v9 exporter: SysID: 1, Domain: 18841857, IP: x.x.x.255
Process_v9: New v9 exporter: SysID: 2, Domain: 2166325505, IP: x.x.x.255

Flow Record: 
  Flags        =              0x00 NETFLOW v9, Unsampled
  Elements     =                 9: 1 3 4 5 6 7 9 11 12 
  size         =               220
  engine type  =               129
  engine ID    =                 1
  export sysid =                 2
  first        =     1731178520000 [2024-11-09 15:55:20.000]
  last         =     1731178520000 [2024-11-09 15:55:20.000]
  received at  =     1731178554478 [2024-11-09 15:55:54.478]
  proto        =                 6 TCP
  tcp flags    =              0x10 ...A....
  src port     =             62636
  dst port     =               443
  src tos      =                 0
  in packets   =                 2
  in bytes     =              2956
  src addr     =  2804:xxxx:acc7:c900:fd70:9972:c1f:a5f0
  dst addr     =  2800:3f0:4001:806::200a
  input        =               142
  output       =               153
  src mask     =                40 acc7:c900::/40
  dst mask     =                48 4001:806:fd70::/48
  fwd status   =               187
  dst tos      =                 1
  direction    =                 1
  biFlow Dir   =              0x00
  end reason   =              0x00
  out packets  =                 0
  out bytes    =                 0
  aggr flows   =                 1
  src vlan     =                 0
  dst vlan     =               447
  src as       =             65530
  dst as       =             15169
  bgp next hop =  fd00:bacb:50:16::2
  ip next hop  =  fd00:bacb:50:16::2
  ip exporter  =   x.x.x.255

Flow Record: 
  Flags        =              0x00 NETFLOW v9, Unsampled
  Elements     =                 9: 1 3 4 5 6 7 9 11 12 
  size         =               220
  engine type  =               129
  engine ID    =                 1
  export sysid =                 2
  first        =     1731178517000 [2024-11-09 15:55:17.000]
  last         =     1731178520000 [2024-11-09 15:55:20.000]
  received at  =     1731178554478 [2024-11-09 15:55:54.478]
  proto        =                 6 TCP
  tcp flags    =              0x10 ...A....
  src port     =                80
  dst port     =             60726
  src tos      =                 0
  in packets   =                 2
  in bytes     =              2582
  src addr     =  2600:9000:277e:7400:2:b65:62c0:93a1
  dst addr     =  2804:xxxx:acca:6d00:b57a:b04b:1f05:5187
  input        =               153
  output       =               142
  src mask     =                48 277e:7400:24::/48
  dst mask     =                40 acca:6d00::/40
  fwd status   =                54
  dst tos      =               237
  direction    =                 0
  biFlow Dir   =              0x00
  end reason   =              0x00
  out packets  =                 0
  out bytes    =                 0
  aggr flows   =                 1
  src vlan     =                 0
  dst vlan     =               445
  src as       =             16509
  dst as       =             65530
  bgp next hop =  fd00:bacb:50:8::2
  ip next hop  =  fd00:bacb:50:8::2
  ip exporter  =   x.x.x.255

File to compare the data collected by the 2 versions: nfcapd.202411091530

ℹ️

Another observation that might be important: from another router, RouterOS v7.16 (MikroTik), the flows are present.

root@flow-upgrade[/var/rr-flows/RS-EMPRESA/2024/11/09]# nfdump -r nfcapd.202411091530  -O flows -A srcip -n 10 
Date first seen         Duration               Src IP Addr   Packets    Bytes      bps    Bpp Flows
2024-11-09 15:29:44.232     00:01:00.000    xxx.xxx.168.14      1599    98730    13164     61   320
2024-11-09 15:29:17.222     00:01:27.010       172.16.16.4      1042    72941     6706     70   209
2024-11-09 15:29:14.152     00:01:29.080       172.16.16.5       853   249607    22416    292   112
2024-11-09 15:29:44.122     00:01:00.110   xxx.xxx.168.218       293    23408     3115     79    68
2024-11-09 15:29:55.142     00:00:42.030   xxx.xxx.168.217        44     1760      334     40    44
2024-11-09 15:28:52.532     00:01:40.510       10.99.98.20     23494   31.8 M    2.5 M   1351    41
2024-11-09 15:29:44.932     00:00:58.990    192.168.254.35        36     2736      371     76    36
2024-11-09 15:29:34.752     00:01:07.800   192.168.254.229       104    32130     3791    308    33
2024-11-09 15:28:48.362     00:01:53.890   192.168.250.138      1213   124119     8718    102    28
2024-11-09 15:30:02.812     00:00:11.130   192.168.254.133       281    35551    25553    126    25
Summary: total flows: 1320, total bytes: 35.8 M, total packets: 42166, avg bps: 2.5 M, avg pps: 363, avg bpp: 848
Time window: 2024-11-09 15:28:48 - 2024-11-09 15:30:44, Duration:    00:01:56.000
Total records processed: 1320, passed: 1320, Blocks skipped: 0, Bytes read: 295808
Sys: 0.0040s User: 0.0040s Wall: 0.0050s flows/second: 262268.2 Runtime: 0.0055s

Apparently, the issue seems to be with the captures coming from the Huawei model.

If you need any further information or any procedure that could help, I am available.

@remontti
Copy link
Author

remontti commented Nov 9, 2024

I forgot to mention how the routers are configured.

ip netstream export version 9 origin-as bgp-nexthop ttl
ip netstream export template sequence-number fixed
ip netstream export index-switch 32
ip netstream as-mode 32
ip netstream timeout active 1
ip netstream timeout inactive 15
ip netstream export template timeout-rate 1
ip netstream export template option timeout-rate 1
ip netstream export template option application-label
ip netstream sampler fix-packets 1024 inbound
ip netstream sampler fix-packets 1024 outbound
ip netstream export source IP_ORIGEM
ip netstream export host FLOW_SERVER 3055

ipv6 netstream export version 9 origin-as bgp-nexthop ttl
ipv6 netstream export template sequence-number fixed
ipv6 netstream export index-switch 32
ipv6 netstream as-mode 32
ipv6 netstream timeout active 1
ipv6 netstream timeout inactive 15
ipv6 netstream export template timeout-rate 1
ipv6 netstream export template option timeout-rate 1
ipv6 netstream sampler fix-packets 1024 inbound
ipv6 netstream sampler fix-packets 1024 outbound
ipv6 netstream export source IP_ORIGEM
ipv6 netstream export host FLOW_SERVER 3055
undo ip netstream export template option sampler
undo ipv6 netstream export template option sampler

interface Virtual-Ethernet0/1/101.408
 description Operadora_1_IPv4
 ip netstream inbound
 ip netstream outbound
interface Virtual-Ethernet0/1/101.409
 description Operadora_1_IPv6
 ipv6 netstream inbound
 ipv6 netstream outbound
interface 40GE0/1/49.2114
 description Operadora_2_IPv4e6
 ip netstream inbound
 ip netstream outbound
 ipv6 netstream inbound
 ipv6 netstream outbound

I captured data from another more up-to-date Huawei router, and the same problem occurs.

VRP (R) software, Version 8.220 (NetEngine 8000 V800R022C00SPC600)
Copyright (C) 2012-2022 Huawei Technologies Co., Ltd.
HUAWEI NetEngine 8000 M8 uptime is 340 days, 1 hour, 56 minutes
Patch Version: V800R022SPH120

@remontti
Copy link
Author

remontti commented Nov 9, 2024

I discovered that using "ipfix" in version 1.7.5 works. But in version 1.7.4, using "ipfix" stops working, with all the flow values ​​being zero.

I think this information may be useful.

@phaag
Copy link
Owner

phaag commented Nov 10, 2024

Could you please send me the file nfcapd.202411091530 of 1.7.4 and 1.7.5 as well as please collect a few minutes worth of pcap traffic sent to the collector and send all that to my email in the authors file.

@remontti
Copy link
Author

Could you please send me the file nfcapd.202411091530 of 1.7.4 and 1.7.5 as well as please collect a few minutes worth of pcap traffic sent to the collector and send all that to my email in the authors file.

nfcapd.202411091530

phaag added a commit that referenced this issue Nov 10, 2024
@phaag
Fix flow counter in sort list. #575
ad15f25
@phaag
Copy link
Owner

phaag commented Nov 10, 2024

Aggregation should work now. Please check.

@remontti
Copy link
Author

Thanks, it seems to be working!

# tcpdump -n udp port 3335 -T cnfp -c 3
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on ens18, link-type EN10MB (Ethernet), snapshot length 262144 bytes
10:26:41.442072 IP x.x.x.x.40000 > 172.16.10.32.3335: NetFlow v9
10:26:45.211791 IP x.x.x.x.40000 > 172.16.10.32.3335: NetFlow v9
10:26:45.211791 IP x.x.x.x.40000 > 172.16.10.32.3335: NetFlow v9
3 packets captured
16 packets received by filter
0 packets dropped by kernel

# nfdump -r nfcapd.202411111023 -O flows -A srcas -n 3
Date first seen         Duration          Src AS   Packets    Bytes      bps    Bpp Flows
2024-11-11 10:22:05.000     00:01:46.000   65530    539648  182.2 M   13.7 M    337   268
2024-11-11 10:22:02.000     00:01:54.000   15169    822272    1.1 G   77.0 M   1334   173
2024-11-11 10:22:40.000     00:01:04.000   32934     43008   22.9 M    2.9 M    531    36
Summary: total flows: 715, total bytes: 3.5 G, total packets: 3.0 M, avg bps: 237.1 M, avg pps: 25783, avg bpp: 1149
Time window: 2024-11-11 10:21:59 - 2024-11-11 10:23:56, Duration:    00:01:57.000
Total records processed: 715, passed: 715, Blocks skipped: 0, Bytes read: 138808
Sys: 0.0078s User: 0.0000s Wall: 0.0039s flows/second: 182961.8 Runtime: 0.0043s

@phaag phaag closed this as completed Nov 12, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@remontti @phaag