Ntopng crashing

[olmanolsztyn]

After last update ntopng crashes.
Before crash there are logs:
17/Aug/2021 11:15:14 [LuaEngine.cpp:560] WARNING: Script failure [/usr/local/share/ntopng/scripts/callbacks/system/second.lua][...local/share/ntopng/scripts/lua/modules/locales_utils.lua:45: attempt to call a string value (local 'chunk')]
17/Aug/2021 11:16:04 [LuaEngine.cpp:560] WARNING: Script failure [/usr/local/share/ntopng/scripts/callbacks/system/timeseries.lua][attempt to call a userdata value]
17/Aug/2021 11:18:01 [LuaEngine.cpp:560] WARNING: Script failure [/usr/local/share/ntopng/scripts/callbacks/system/notifications.lua][/usr/local/share/ntopng/scripts/lua/modules/dkjson.lua:397: bad argument #2 to 'strfind' (string expected, got no value)]
17/Aug/2021 11:18:05 [LuaEngine.cpp:560] WARNING: Script failure [/usr/local/share/ntopng/scripts/callbacks/system/second.lua][/usr/local/share/ntopng/scripts/lua/modules/i18n/i18n.lua:56: i18n.load requires a non-empty string or table on its en.internals.flow_calls_stats. Got flow.lua Calls Stats (a string value).]
17/Aug/2021 11:18:06 [LuaEngine.cpp:560] WARNING: Script failure [/usr/local/share/ntopng/scripts/callbacks/system/housekeeping.lua][attempt to index a function value]
17/Aug/2021 11:19:01 [LuaEngine.cpp:560] WARNING: Script failure [/usr/local/share/ntopng/scripts/callbacks/system/second.lua][...local/share/ntopng/scripts/lua/modules/locales_utils.lua:45: attempt to call a string value (local 'chunk')]
17/Aug/2021 11:24:07 [LuaEngine.cpp:560] WARNING: Script failure [/usr/local/share/ntopng/scripts/callbacks/system/second.lua][attempt to concatenate a userdata value]
17/Aug/2021 11:26:06 [LuaEngine.cpp:560] WARNING: Script failure [/usr/local/share/ntopng/scripts/callbacks/system/second.lua][/usr/local/share/ntopng/scripts/callbacks/system/second.lua:29: module 'ts_utils_core' not found:@/usr/local/share/ntopng/scripts/lua/modules/timeseries/ts_utils_core.lua]
17/Aug/2021 11:37:01 [LuaEngine.cpp:560] WARNING: Script failure [/usr/local/share/ntopng/scripts/callbacks/system/timeseries.lua][...local/share/ntopng/scripts/lua/modules/locales_utils.lua:45: attempt to call a string value (local 'chunk')]
17/Aug/2021 11:39:03 [LuaEngine.cpp:560] WARNING: Script failure [/usr/local/share/ntopng/scripts/callbacks/system/second.lua][/usr/local/share/ntopng/scripts/lua/modules/i18n/i18n.lua:23: bad argument #3 to 'gsub' (number expected, got no value)]
17/Aug/2021 11:40:10 [LuaEngine.cpp:560] WARNING: Script failure [/usr/local/share/ntopng/scripts/callbacks/system/second.lua][/usr/local/share/ntopng/scripts/lua/modules/lua_utils.lua:23: module 'lua_trace' not found:@/usr/local/share/ntopng/scripts/lua/modules/lua_trace.lua]
17/Aug/2021 11:40:16 [LuaEngine.cpp:560] WARNING: Script failure [/usr/local/share/ntopng/scripts/callbacks/system/timeseries.lua][/usr/local/share/ntopng/scripts/lua/modules/ntop_utils.lua:275: object length is not an integer]
17/Aug/2021 11:40:21 [LuaEngine.cpp:560] WARNING: Script failure [/usr/local/share/ntopng/scripts/callbacks/system/second.lua][.../ntopng/scripts/lua/modules/timeseries/ts_utils_core.lua:139: module 'rrd' not found:@/usr/local/share/ntopng/scripts/lua/modules/timeseries/drivers/rrd.lua]

version:
Version: 4.3.210817 [Enterprise/Professional build]
Built on: FreeBSD 12.2
Edition: Enterprise M

[olmanolsztyn]

After restart in addition there is info like on the screenshot:

[olmanolsztyn]

Another log before crash:
17/Aug/2021 12:55:59 [get_check_config.lua:44] ERROR: Unknown user script: syn_flood_victim

[olmanolsztyn]

Another logs before crash:
17/Aug/2021 15:13:00 [LuaEngine.cpp:560] WARNING: Script failure [/usr/local/share/ntopng/scripts/callbacks/system/discover.lua][...ocal/share/ntopng/scripts/lua/modules/discover_utils.lua:288: attempt to concatenate a nil value (local 'ifid')]
17/Aug/2021 15:13:00 [LuaEngine.cpp:560] WARNING: Script failure [/usr/local/share/ntopng/scripts/callbacks/system/timeseries.lua][/usr/local/share/ntopng/scripts/lua/modules/ntop_utils.lua:279: bad argument #2 to 'sub' (number expected, got no value)]
17/Aug/2021 15:14:00 [LuaEngine.cpp:560] WARNING: Script failure [/usr/local/share/ntopng/scripts/callbacks/system/discover.lua][...ocal/share/ntopng/scripts/lua/modules/discover_utils.lua:288: attempt to concatenate a nil value (local 'ifid')]

[MatteoBiscosi]

hi @olmanolsztyn is ntopng going to crash just randomly after a restart or do you do something specific before it crashes?
Regarding the last screen you added, it's correct this way, in this way users are able to understand when the data are going to be shown up in the web gui after a restart

[olmanolsztyn]

It crashed randomly.
Yesterday i disabled some alert checks and restart server. Didn't crash so far.
Didn't make any ntopng configuration changes in ntopng.conf.

[MatteoBiscosi]

alright, let me know if it crashes again. Do you remember which are the checks you disabled?

[olmanolsztyn]

As i remember it was Remote Connection, SYN Flood Victim Alert, Blacklisted Flow, Remote to Local Insecure Protocol.

[MatteoBiscosi]

Sorry @olmanolsztyn but I was unable to reproduce your crash, please let me know first if with those alerts disabled the problem still remains or not and if it's going to crash again please send me the entire logs of ntopng. If you prefer, you can drop an email to my address, [email protected]

[MatteoBiscosi]

@olmanolsztyn if you have the possibility, could you please re-add them one by one and let me know with which checks enabled it crashes?

[olmanolsztyn]

Server crashed again and i got log:
19/Aug/2021 05:59:01 [LuaEngine.cpp:560] WARNING: Script failure [/usr/local/share/ntopng/scripts/callbacks/system/timeseries.lua][attempt to index a function value]

[olmanolsztyn]

Suddenly ntopng stopped storing flows after i went System > Health > Interface > tcp://127.0.0.1:5000c

[simonemainardi]

@olmanolsztyn if you have the possibility, could you please re-add them one by one and let me know with which checks enabled it crashes?

@olmanolsztyn did you enable checks as suggested by matteo to trigger the crash?

[olmanolsztyn]

I did not make it. Ntopng process stopped working eariler.

[olmanolsztyn]

In addition i updated today ntopng to ntopng Enterprise M v.5.1.210819 (FreeBSD 12.2) version.

[MatteoBiscosi]

Hi @olmanolsztyn do you perhaps have the top sites enabled? (settings->applications->top visited sites)

[olmanolsztyn]

Yes i have enabled.

[MatteoBiscosi]

alright soon i'm going to release a fix, those were probably the crashing problem, I happened to encounter it randomly before, so please be patience, the fix is already on testing, I'll push it really soon so in the afternoon you should have the new build, please update and let me know if it continues crashing after the new build is released

[MatteoBiscosi]

Hi @olmanolsztyn any news?

[olmanolsztyn]

Looks like problem is solved. Didn't crash so far.

[MatteoBiscosi]

Great! I'm going to close the ticket, please re-open it or contact us if the issue persists

[olmanolsztyn]

Ntopng after update and enabling top visited sites crashed again.

[olmanolsztyn]

Another crash with disable top visited sites
Below log before crash:
28/Aug/2021 10:32:01 [LuaEngine.cpp:560] WARNING: Script failure [/usr/local/share/ntopng/scripts/callbacks/system/second.lua][attempt to index a function value]