64 bit counters for IN_BYTES and OUT_BYTES

[cosidius]

nfcapd cant handle the 8*8bit, 64 bit uint from Fortigate for "Field Type 1" and "23" correct so the couter values is wrong. Its also saves the data wrong on disk so if one uses nfreplay the values sent out are faulty.

Below is the output I get with the exact same flow in Cisco and Fortigate. The only difference is that Fortigate uses 64 bit counters for bytes.

Fortigate

./nfdump -o raw -R /mnt/netflows-fgc/2016/03/18/13/nfcapd.201603181300 "not src net 130.240/16 and src host 77.218.251.180 and dst host 130.240.19.2 and port 1144"

Flow Record:
Flags = 0x86 FLOW, Sampled
export sysid = 1
size = 60
first = 1458304467 [2016-03-18 13:34:27]
last = 1458304467 [2016-03-18 13:34:27]
msec_first = 960
msec_last = 960
src addr = 77.218.251.180
dst addr = 130.240.19.2
src port = 1144
dst port = 53
fwd status = 64
tcp flags = 0x00 ......
proto = 17 UDP
(src)tos = 0
(in)packets = 6963
(in)bytes = 752004
input = 50
output = 89

Summary: total flows: 1, total bytes: 752004, total packets: 6963, avg bps: 0, avg pps: 0, avg bpp: 0
Time window: 2016-02-02 14:37:50 - 2016-03-18 13:59:59
Total flows processed: 4612764, Blocks skipped: 0, Bytes read: 278971320
Sys: 1.385s flows/second: 3328619.3 Wall: 2.240s flows/second: 2058970.9

Cisco

./nfdump -o raw -R /mnt/netflows/2016/03/18/13/nfcapd.201603181300 "not src net 130.240/16 and src host 77.218.251.180 and dst host 130.240.19.2 and port 1144"

Flow Record:
Flags = 0x06 FLOW, Unsampled
export sysid = 4
size = 60
first = 1458304468 [2016-03-18 13:34:28]
last = 1458304468 [2016-03-18 13:34:28]
msec_first = 506
msec_last = 506
src addr = 77.218.251.180
dst addr = 130.240.19.2
src port = 1144
dst port = 53
fwd status = 0
tcp flags = 0x00 ......
proto = 17 UDP
(src)tos = 0
(in)packets = 1
(in)bytes = 108
input = 155
output = 541

Summary: total flows: 1, total bytes: 108, total packets: 1, avg bps: 0, avg pps: 0, avg bpp: 0
Time window: 2016-03-18 12:27:53 - 2016-03-18 13:59:57
Total flows processed: 4962501, Blocks skipped: 0, Bytes read: 300312984
Sys: 2.209s flows/second: 2245818.0 Wall: 3.816s flows/second: 1300159.2

Ex. Template Cisco
Sender = 533 FlowSetID = 0 Length = 84
Template ID = 256 Field count = 19
Field Type = 21 , Field Length = 4
Field Type = 22 , Field Length = 4
Field Type = 1 , Field Length = 4
Field Type = 2 , Field Length = 4
Field Type = 10 , Field Length = 2
Field Type = 14 , Field Length = 2
Field Type = 8 , Field Length = 4
Field Type = 12 , Field Length = 4
Field Type = 4 , Field Length = 1
Field Type = 5 , Field Length = 1
Field Type = 7 , Field Length = 2
Field Type = 11 , Field Length = 2
Field Type = 48 , Field Length = 1
Field Type = 51 , Field Length = 1
Field Type = 15 , Field Length = 4
Field Type = 13 , Field Length = 1
Field Type = 9 , Field Length = 1
Field Type = 6 , Field Length = 1
Field Type = 61 , Field Length = 1

Ex. Template Fortigate
Sender = 1 FlowSetID = 0 Length = 972
Template ID = 257 Field count = 17
Field Type = 1 , Field Length = 8
Field Type = 23 , Field Length = 8
Field Type = 2 , Field Length = 4
Field Type = 24 , Field Length = 4
Field Type = 22 , Field Length = 4
Field Type = 21 , Field Length = 4
Field Type = 7 , Field Length = 2
Field Type = 11 , Field Length = 2
Field Type = 10 , Field Length = 2
Field Type = 14 , Field Length = 2
Field Type = 4 , Field Length = 1
Field Type = 95 , Field Length = 4
Field Type = 65 , Field Length = 2
Field Type = 89 , Field Length = 1
Field Type = 136 , Field Length = 1
Field Type = 8 , Field Length = 4
Field Type = 12 , Field Length = 4

[phaag]

nfdump handles 8bytes (64bit) counters already for years. INternally nfdump uses only 64bit counters.
If the counters are wrong, this mus be something different. Please send me a pcap captured at the collector side to check.

[cosidius]

Hi,

Here you have a pcap file and the debug output from my perl script parsing the same pcap file.

If you want to make it easy for you to see the difference when 64 bit fields for IN_Bytes and OUT_bytes is used use "dst host 130.240.19.2" and if source starts with 130.240.x.x then it is processed by Fortigate. Else it processed by Cisco with normal 32bit counters.

130.240.19.2 is one of our DNS servers

---

Från: Peter Haag [email protected]
Skickat: den 6 maj 2016 14:18
Till: phaag/nfdump
Kopia: Thomas Nilsson; Author
Ämne: {Disarmed} Re: [phaag/nfdump] 64 bit counters for IN_BYTES and OUT_BYTES (#22)

nfdump handles 8bytes (64bit) counters already for years. INternally nfdump uses only 64bit counters.
If the counters are wrong, this mus be something different. Please send me a pcap captured at the collector side to check.

You are receiving this because you authored the thread.
Reply to this email directly or view it on GitHubhttps://github.com//issues/22#issuecomment-217425141

[cosidius]

PecapAndDebug.zip

[cosidius]

Hi,

Are you looking into this issue even when you closed it before I had the time to reply, or do I need to open a new ticket?

[cosidius]

Could be that Fortigate don’t send TCP flags in their export. If this field is missing how do your code cope with it?