Uncompress_Block_LZO() error decompression failed in nffile.c

[mkoelewijn]

Getting lots of these errors with nfdump version 1.7.1 when i do a query like this:

nfdump -M /data/nfsen/profiles-data/malicious-inra2/in:out -T -R 2023/01/08/nfcapd.202301081500:2023/01/09/nfcapd.202301090625 -n 10 -s ip/flows

Uncompress_Block_LZO() error decompression failed in nffile.c line 212: LZO error: -6

Any advice how to get rid of these errors?

[phaag]

Hmm .. are you sure, your files have been written correctly? I would suspect, that you possibly were short of disk space and files could not have been written, which results in corrupt files.

[mkoelewijn]

Data is not a problem: /dev/nvme0n1 7.0T 1.7T 4.9T 26% /data

I was not completely clear with my ticket. I used 1.7.0.1 for a few weeks now and with the data i captured with that i get those errors but i would still get the results printed. Now since a few hours i use 1.7.1 and i don't see the errors anymore. Will monitor this for some longer period to see what happens.

Some other weird thing i get is whenever i restart my nfcapd processes the data is not correct. For the first 10 minutes it shows 474598.4 Tb/s (all data) for my stream while after 10 minutes it normalises to the real throughput of 22.8 Gb/s (all data). The flows count is good from the start, i use a 1/100 sampling rate.

[mkoelewijn]

Moreover, i just tested some query's on the live profile and with a separate profile (malicious-inra2). The errors only occur in the malicious-inra2 profile and not in the live profile. But again, don't see this problem yet in the last couple of hours, so will monitor that for longer to see if i get it again.

[phaag]

The nffile code has not changed since Sept last year. So there is nothing, I would suspect. nffile is one of the very central parts, which are used for each run. Moreover, if it occurs only on some profiles, this does not make any sense to me.

As of restart nfcapd, there may be sampling involved, which your exporter sends only in periodic intervals.

[mkoelewijn]

Don't see the decompression errors anymore, ticket closed.

[wallacemariadeandrade]

Hi there, i'm having this same problem with many files. Take a look:

root@nfsen:/var/nfsen/profiles-data/AKAMAI-EGRESS/RT_BGP/2023/04/24# nfdump -r nfcapd.202304240640
Date first seen             Duration     Proto      Src IP Addr(..):Port          Dst IP Addr(..):Port   Packets    Bytes Flows
Uncompress_Block_LZO() error decompression failed in nffile.c line 215: LZO error: -6
Unable to read appendix block of file: nfcapd.202304240640
Uncompress_Block_LZO() error decompression failed in nffile.c line 215: LZO error: -6
No matching flows
Summary: total flows: 0, total bytes: 0, total packets: 0, avg bps: 0, avg pps: 0, avg bpp: 0
Time window: <unknown>
Total flows processed: 0, passed: 0, Blocks skipped: 0, Bytes read: 0
Sys: 0.4349s User: 1.5183s Wall: 0.0005s flows/second: 0.0 Runtime: 0.0006s
root@nfsen:/var/nfsen/profiles-data/AKAMAI-EGRESS/RT_BGP/2023/04/24# nfdump -r nfcapd.202304240645
Date first seen             Duration     Proto      Src IP Addr(..):Port          Dst IP Addr(..):Port   Packets    Bytes Flows
Uncompress_Block_LZO() error decompression failed in nffile.c line 215: LZO error: -6
Unable to read appendix block of file: nfcapd.202304240645
Uncompress_Block_LZO() error decompression failed in nffile.c line 215: LZO error: -6
No matching flows
Summary: total flows: 0, total bytes: 0, total packets: 0, avg bps: 0, avg pps: 0, avg bpp: 0
Time window: <unknown>
Total flows processed: 0, passed: 0, Blocks skipped: 0, Bytes read: 0
Sys: 0.4102s User: 1.5411s Wall: 0.0006s flows/second: 0.0 Runtime: 0.0007s
root@nfsen:/var/nfsen/profiles-data/AKAMAI-EGRESS/RT_BGP/2023/04/24# nfdump -r nfcapd.202304240650
Date first seen             Duration     Proto      Src IP Addr(..):Port          Dst IP Addr(..):Port   Packets    Bytes Flows
Uncompress_Block_LZO() error decompression failed in nffile.c line 215: LZO error: -6
No matching flows
Summary: total flows: 0, total bytes: 0, total packets: 0, avg bps: 0, avg pps: 0, avg bpp: 0
Time window: <unknown>
Total flows processed: 0, passed: 0, Blocks skipped: 0, Bytes read: 0
Sys: 0.4211s User: 1.5269s Wall: 0.0005s flows/second: 0.0 Runtime: 0.0006s
root@nfsen:/var/nfsen/profiles-data/AKAMAI-EGRESS/RT_BGP/2023/04/24# 

And there's a lot of free space in disk:

root@nfsen:/var/nfsen/profiles-data/AKAMAI-EGRESS/RT_BGP/2023/04/24# df -h
Filesystem      Size  Used Avail Use% Mounted on
udev            2.0G     0  2.0G   0% /dev
tmpfs           394M  632K  393M   1% /run
/dev/sda1        97G  3.8G   89G   5% /
tmpfs           2.0G     0  2.0G   0% /dev/shm
tmpfs           5.0M     0  5.0M   0% /run/lock
tmpfs           394M     0  394M   0% /run/user/0

I have other profiles but it's not occurring with them all. Nfdump version is nfdump: Version: 1.7.2-62ca665 NSEL-NEL, Date: 2023-04-23 14:29:32 +0200

Is there something I can do?

[phaag]

Could you pleas send my by email to the address in the AUTHORS file:

From profile live: file nfcapd.202304240640
From profile AKAMAI-EGRESS/RT_BGP : nfcapd.202304240640
The filter for the profile AKAMAI-EGRESS/RT_BGP

Thanks!

[wallacemariadeandrade]

Unfortunately I have erased AKAMAI profile, but I have other with same problem. Take a look:

root@hexaflow:/var/nfsen/profiles-data/FACEBOOK-INGRESS/RT_BGP/2023/04/29# nfdump -r nfcapd.202304290730 
Date first seen             Duration     Proto      Src IP Addr(..):Port          Dst IP Addr(..):Port   Packets    Bytes Flows
Uncompress_Block_LZO() error decompression failed in nffile.c line 215: LZO error: -6
No matching flows
Summary: total flows: 0, total bytes: 0, total packets: 0, avg bps: 0, avg pps: 0, avg bpp: 0
Time window: 2023-04-29 07:28:59 - 2023-04-29 07:34:58
Total flows processed: 0, passed: 0, Blocks skipped: 0, Bytes read: 0

I sent you files from live and facebook profiles, together with the filter used!
Thank you so much!

[wallacemariadeandrade]

Last commit solved the problem! I'm running for 2 days and I don't see any error :D

Thank you so much!

[phaag]

The only affected binary was nfprofile. nfcapd always collected correctly.
Thanks for sorting thus out!