ndpiReader crashes with latest git repo code when run with '-c mining_hosts.txt'

[rkerur]

No description provided.

[rkerur]

gdb ./ndpiReader
GNU gdb (Ubuntu 7.11.1-0ubuntu1~16.5) 7.11.1
Copyright (C) 2016 Free Software Foundation, Inc.
License GPLv3+: GNU GPL version 3 or later http://gnu.org/licenses/gpl.html
This is free software: you are free to change and redistribute it.
There is NO WARRANTY, to the extent permitted by law. Type "show copying"
and "show warranty" for details.
This GDB was configured as "x86_64-linux-gnu".
Type "show configuration" for configuration details.
For bug reporting instructions, please see:
http://www.gnu.org/software/gdb/bugs/.
Find the GDB manual and other documentation resources online at:
http://www.gnu.org/software/gdb/documentation/.
For help, type "help".
Type "apropos word" to search for commands related to "word"...
Reading symbols from ./ndpiReader...done.
(gdb) run -i ../../pcap_files/webex.pcap -v 3 -c mining_hosts.txt
Starting program: /home/vagrant/nDPI/example/ndpiReader -i ../../pcap_files/webex.pcap -v 3 -c mining_hosts.txt
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".

---

• NOTE: This is demo app to show some nDPI features.
• In this demo we have implemented only some basic features
• just to show you what you can do with the library. Feel
• free to extend it and send us the patches for inclusion

---

Using nDPI (2.9.0-1877-6a22bee) [1 thread(s)]
Reading packets from pcap file ../../pcap_files/webex.pcap...
[Category] Loading cnhv.co 99
[Category] Loading coin-hive.com 99
[Category] Loading coinhive.com 99
[Category] Loading authedmine.com 99
[Category] Loading gus.host 99
[Category] Loading load.jsecoin.com 99
[Category] Loading miner.pr0gramm.com 99
[Category] Loading minemytraffic.com 99
[Category] Loading ppoi.org 99
[Category] Loading projectpoi.com 99
[Category] Loading crypto-loot.com 99
[Category] Loading coinerra.com 99
[Category] Loading coin-have.com 99
[Category] Loading minero.pw 99
[Category] Loading minero-proxy-01.now.sh 99
[Category] Loading minero-proxy-02.now.sh 99
[Category] Loading minero-proxy-03.now.sh 99
[Category] Loading api.inwemo.com 99
[Category] Loading rocks.io 99
[Category] Loading adminer.com 99
[Category] Loading ad-miner.com 99
[Category] Loading jsccnn.com 99
[Category] Loading jscdndel.com 99
[Category] Loading coinhiveproxy.com 99
[Category] Loading coinblind.com 99
[Category] Loading coinnebula.com 99
[Category] Loading monerominer.rocks 99
[Category] Loading cdn.cloudcoins.co 99
[Category] Loading coinlab.biz 99
[Category] Loading go.megabanners.cf 99
[Category] Loading baiduccdn1.com 99
[Category] Loading wsp.marketgid.com 99
[Category] Loading papoto.com 99
[Category] Loading flare-analytics.com 99
[Category] Loading www.sparechange.io 99
[Category] Loading static.sparechange.io 99
[Category] Loading miner.nablabee.com 99
[Category] Loading m.anyfiles.ovh 99
[Category] Loading www.freecontent.bid 99
[Category] Loading www.freecontent.loan 99
[Category] Loading www.freecontent.win 99
[Category] Loading www.cryptonoter.com 99
[Category] Loading www.mutuza.win 99
[Category] Loading cryweb.github.io 99
[Category] Loading crywebber.github.io 99
[Category] Loading crypto-webminer.com 99
[Category] Loading cdn.adless.io 99
[Category] Loading hegrinhar.com 99
[Category] Loading verresof.com 99
[Category] Loading hemnes.win 99
[Category] Loading tidafors.xyz 99
[Category] Loading moneone.ga 99
[Category] Loading plexcoin.info 99
[Category] Loading www.monkeyminer.net 99
[Category] Loading go2.mercy.ga 99
[Category] Loading coinpirate.cf 99
[Category] Loading d.cpufan.club 99
[Category] Loading krb.devphp.org.ua 99
[Category] Loading nfwebminer.com 99
[Category] Loading cfcdist.gdn 99
[Category] Loading node.cfcdist.gdn 99
[Category] Loading webxmr.com 99
[Category] Loading xmr.mining.best 99
[Category] Loading webminepool.com 99
[Category] Loading webminepool.tk 99
[Category] Loading hive.tubetitties.com 99
[Category] Loading playerassets.info 99
[Category] Loading tokyodrift.ga 99
[Category] Loading webassembly.stream 99
[Category] Loading okeyletsgo.ml 99
[Category] Loading candid.zone 99
[Category] Loading webmine.pro 99
[Category] Loading andlache.com 99
[Category] Loading bablace.com 99
[Category] Loading bewaslac.com 99
[Category] Loading biberukalap.com 99
[Category] Loading bowithow.com 99
[Category] Loading butcalve.com 99
[Category] Loading evengparme.com 99
[Category] Loading gridiogrid.com 99
[Category] Loading hatcalter.com 99
[Category] Loading kedtise.com 99
[Category] Loading ledinund.com 99
[Category] Loading nathetsof.com 99
[Category] Loading renhertfo.com 99
[Category] Loading rintindown.com 99
[Category] Loading sparnove.com 99
[Category] Loading witthethim.com 99
[Category] Loading 1q2w3.fun 99
[Category] Loading 1q2w3.me 99
[Category] Loading cryptoloot.pro 99
[Category] Loading bjorksta.men 99
[Category] Loading crypto.csgocpu.com 99
[Category] Loading noblock.pro 99
[Category] Loading miner.cryptobara.com 99
[Category] Loading digger.cryptobara.com 99
[Category] Loading dev.cryptobara.com 99
[Category] Loading reservedoffers.club 99
[Category] Loading mine.torrent.pw 99
[Category] Loading host.d-ns.ga 99
[Category] Loading abc.pema.cl 99
[Category] Loading mine.nahnoji.cz 99
[Category] Loading webmine.cz 99
[Category] Loading intactoffers.club 99
[Category] Loading analytics.blue 99
[Category] Loading smectapop12.pl 99
[Category] Loading berserkpl.net.pl 99
[Category] Loading hodlers.party 99
[Category] Loading hodling.faith 99
[Category] Loading chainblock.science 99
[Category] Loading minescripts.info 99
[Category] Loading cdn.minescripts.info 99
[Category] Loading miner.nablabee.com 99
[Category] Loading wss.nablabee.com 99
[Category] Loading clickwith.bid 99
[Category] Loading dronml.ml 99
[Category] Loading niematego.tk 99
[Category] Loading tulip18.com 99
[Category] Loading p.estream.to 99
[Category] Loading didnkinrab.com 99
[Category] Loading ledhenone.com 99
[Category] Loading losital.ru 99
[Category] Loading mebablo.com 99
[Category] Loading moonsade.com 99
[Category] Loading nebabrop.com 99
[Category] Loading pearno.com 99
[Category] Loading rintinwa.com 99
[Category] Loading willacrit.com 99
[Category] Loading www2.adfreetv.ch 99
[Category] Loading minr.pw 99
[Category] Loading new.minr.pw 99
[Category] Loading test.minr.pw 99
[Category] Loading staticsfs.host 99
[Category] Loading cdn-code.host 99
[Category] Loading g-content.bid 99
[Category] Loading ad.g-content.bid 99
[Category] Loading cdn.static-cnt.bid 99
[Category] Loading cnt.statistic.date 99
[Category] Loading cdn.jquery-uim.download 99
[Category] Loading cdn-jquery.host 99
[Category] Loading p1.interestingz.pw 99
[Category] Loading kippbeak.cf 99
[Category] Loading pasoherb.gq 99
[Category] Loading axoncoho.tk 99
[Category] Loading depttake.ga 99
[Category] Loading flophous.cf 99
[Category] Loading pr0gram.org 99
[Category] Loading authedmine.eu 99
[Category] Loading www.monero-miner.com 99
[Category] Loading www.datasecu.download 99
[Category] Loading www.jquery-cdn.download 99
[Category] Loading www.etzbnfuigipwvs.ru 99
[Category] Loading www.terethat.ru 99
[Category] Loading freshrefresher.com 99
[Category] Loading api.pzoifaum.info 99
[Category] Loading ws.pzoifaum.info 99
[Category] Loading api.bhzejltg.info 99
[Category] Loading ws.bhzejltg.info 99
[Category] Loading d.cfcnet.top 99
[Category] Loading vip.cfcnet.top 99
[Category] Loading eu.cfcnet.top 99
[Category] Loading as.cfcnet.top 99
[Category] Loading us.cfcnet.top 99
[Category] Loading eu.cfcdist.loan 99
[Category] Loading as.cfcdist.loan 99
[Category] Loading us.cfcdist.loan 99
[Category] Loading gustaver.ddns.net 99
[Category] Loading worker.salon.com 99
[Category] Loading s2.appelamule.com 99
[Category] Loading mepirtedic.com 99
[Category] Loading cdn.streambeam.io 99
[Category] Loading adzjzewsma.cf 99
[Category] Loading ffinwwfpqi.gq 99
[Category] Loading ininmacerad.pro 99
[Category] Loading mhiobjnirs.gq 99
[Category] Loading open-hive-server-1.pp.ua 99
[Category] Loading pool.hws.ru 99
[Category] Loading pool.etn.spacepools.org 99
[Category] Loading api.aalbbh84.info 99
[Category] Loading www.aymcsx.ru 99
[Category] Loading aeros01.tk 99
[Category] Loading aeros02.tk 99
[Category] Loading aeros03.tk 99
[Category] Loading aeros04.tk 99
[Category] Loading aeros05.tk 99
[Category] Loading aeros06.tk 99
[Category] Loading aeros07.tk 99
[Category] Loading aeros08.tk 99
[Category] Loading aeros09.tk 99
[Category] Loading aeros10.tk 99
[Category] Loading aeros11.tk 99
[Category] Loading aeros12.tk 99
[Category] Loading npcdn1.now.sh 99
[Category] Loading mxcdn2.now.sh 99
[Category] Loading sxcdn6.now.sh 99
[Category] Loading mxcdn1.now.sh 99
[Category] Loading sxcdn02.now.sh 99
[Category] Loading sxcdn4.now.sh 99
[Category] Loading jqcdn2.herokuapp.com 99
[Category] Loading sxcdn1.herokuapp.com 99
[Category] Loading sxcdn5.herokuapp.com 99
[Category] Loading wpcdn1.herokuapp.com 99
[Category] Loading jqcdn01.herokuapp.com 99
[Category] Loading jqcdn03.herokuapp.com 99
[Category] Loading 1q2w3.website 99
[Category] Loading video.videos.vidto.me 99
[Category] Loading video.streaming.estream.to 99
[Category] Loading eth-pocket.de 99
[Category] Loading xvideosharing.site 99
[Category] Loading bestcoinsignals.com 99
[Category] Loading eucsoft.com 99
[Category] Loading traviilo.com 99
[Category] Loading wasm24.ru 99
[Category] Loading xmr.cool 99
[Category] Loading api.netflare.info 99
[Category] Loading cdnjs.cloudflane.com 99
[Category] Loading www.cloudflane.com 99
[Category] Loading hide.ovh 99
[Category] Loading graftpool.ovh 99
[Category] Loading encoding.ovh 99
[Category] Loading altavista.ovh 99
[Category] Loading scaleway.ovh 99
[Category] Loading nexttime.ovh 99
[Category] Loading never.ovh 99
[Category] Loading 2giga.download 99
[Category] Loading support.2giga.link 99
[Category] Loading webminerpool.com 99
[Category] Loading minercry.pt 99
[Category] Loading adplusplus.fr 99
[Category] Loading azvjudwr.info 99
[Category] Loading jroqvbvw.info 99
[Category] Loading jyhfuqoh.info 99
[Category] Loading kdowqlpt.info 99
[Category] Loading xbasfbno.info 99
[Category] Loading 1beb2a44.space 99
[Category] Loading 300ca0d0.space 99
[Category] Loading 310ca263.space 99
[Category] Loading 320ca3f6.space 99
[Category] Loading 330ca589.space 99
[Category] Loading 340ca71c.space 99
[Category] Loading 360caa42.space 99
[Category] Loading 370cabd5.space 99
[Category] Loading 3c0cb3b4.space 99
[Category] Loading 3d0cb547.space 99
[Category] Loading imageshack.us 99
[Category] Loading img265.imageshack.us 99

Program received signal SIGSEGV, Segmentation fault.
strlen () at ../sysdeps/x86_64/strlen.S:106
106 ../sysdeps/x86_64/strlen.S: No such file or directory.
(gdb) bt
#0 strlen () at ../sysdeps/x86_64/strlen.S:106
#1 0x000000000045c116 in ue2::addExpression (ng=..., index=index@entry=266, expression=0x0, flags=0, ext=0x0, id=0) at /home/vagrant/hyperscan/src/compiler/compiler.cpp:284
#2 0x0000000000459689 in ue2::hs_compile_multi_int (expressions=expressions@entry=0x139eff0, flags=flags@entry=0x0, ids=ids@entry=0x139f8f0, ext=ext@entry=0x0, elements=elements@entry=286, mode=mode@entry=1, platform=0x0,
db=0x1396e90, comp_error=0x7fffffffdea0, g=...) at /home/vagrant/hyperscan/src/hs.cpp:239
#3 0x000000000045a46f in hs_compile_multi (expressions=0x139eff0, flags=flags@entry=0x0, ids=0x139f8f0, elements=286, mode=mode@entry=1, platform=platform@entry=0x0, db=0x1396e90, error=0x7fffffffdea0)
at /home/vagrant/hyperscan/src/hs.cpp:314
#4 0x0000000000425ad5 in hyperscan_load_patterns (hs=0x1396e90, num_patterns=, expressions=, ids=) at ndpi_main.c:555
#5 0x000000000042891c in ndpi_enable_loaded_categories (ndpi_str=0x10b5bb0) at ndpi_main.c:4354
#6 0x000000000040c1c7 in setupDetection (thread_id=, pcap_handle=) at ndpiReader.c:1916
#7 0x00000000004186dd in test_lib () at ndpiReader.c:3402
#8 0x000000000040a21a in main (argc=, argv=) at ndpiReader.c:4203
(gdb)

[lucaderi]

I don't compile nDPI with HyperScan since some time. I believe the bug is there

[rkerur]

Bug was introduced recently. Hyperscan+ mining_hosts worked fine until now.

I commented out following code in ndpi_content_match.c.inc and crash does not happen.

static ndpi_category_match category_match[] = {
/*
{ ".edgecastcdn.net", "egdecastcdn" TLD, NDPI_PROTOCOL_CATEGORY_MEDIA },
{ ".hwcdn.net", "hwcdn" TLD, NDPI_PROTOCOL_CATEGORY_MEDIA },
{ ".llnwd.net", "llnwd" TLD, NDPI_PROTOCOL_CATEGORY_MEDIA },
{ ".llns.net", "llns" TLD, NDPI_PROTOCOL_CATEGORY_MEDIA },
{ ".fastly.net", "fastly" TLD, NDPI_PROTOCOL_CATEGORY_MEDIA },
{ ".akamaiedge.net", "akamaiedge" TLD, NDPI_PROTOCOL_CATEGORY_MEDIA },
{ ".vultr.com", "vultr" TLD, NDPI_PROTOCOL_CATEGORY_CLOUD },
{ "baidu.com", "baidu" TLD, NDPI_PROTOCOL_CATEGORY_WEB },
{ "icq.com", "icq" TLD, NDPI_PROTOCOL_CATEGORY_CHAT },
{ "quickplay.com", "quickplay" TLD, NDPI_PROTOCOL_CATEGORY_STREAMING },
{ ".iqiyi.com", "\.iqiyi" TLD, NDPI_PROTOCOL_CATEGORY_STREAMING },
{ ".qiyi.com", "\.qiyi" TLD, NDPI_PROTOCOL_CATEGORY_STREAMING },
{ ".71.am", "\.71" TLD, NDPI_PROTOCOL_CATEGORY_STREAMING },
{ ".qiyipic.com", "\.qiyipic" TLD, NDPI_PROTOCOL_CATEGORY_STREAMING },
{ ".1kxun.", "\.1kxun\.", NDPI_PROTOCOL_CATEGORY_STREAMING },
{ "tcad.wedolook.com", "tcad\.wedolook" TLD, NDPI_PROTOCOL_CATEGORY_STREAMING },
{ ".rapidvideo.com", "\.rapidvideo" TLD, NDPI_PROTOCOL_CATEGORY_STREAMING },
{ ".playercdn.net", "\.playercdn" TLD, NDPI_PROTOCOL_CATEGORY_STREAMING },
{ "showmax.com", "showmax" TLD, NDPI_PROTOCOL_CATEGORY_STREAMING },
{ "showmax.akamaized.net", "showmax\.akamaized" TLD, NDPI_PROTOCOL_CATEGORY_STREAMING },
{ NULL, NULL, 0 } */
};

[rkerur]

Following changes fixes the problem.

git diff src/lib/ndpi_main.c
diff --git a/src/lib/ndpi_main.c b/src/lib/ndpi_main.c
index feddae6..af125a3 100644
--- a/src/lib/ndpi_main.c
+++ b/src/lib/ndpi_main.c
@@ -4264,7 +4264,6 @@ void ndpi_load_ip_category(struct ndpi_detection_module_struct *ndpi_str,
*/
int ndpi_load_hostname_category(struct ndpi_detection_module_struct *ndpi_str,
char *name, ndpi_protocol_category_t category) {

• AC_PATTERN_t ac_pattern;

  if(name == NULL)
  return(-1);
  @@ -4273,8 +4272,6 @@ int ndpi_load_hostname_category(struct ndpi_detection_module_struct *ndpi_str,
  printf("===> %s() Loading %s as %u\n", FUNCTION, name, category);
  #endif

• memset(&ac_pattern, 0, sizeof(ac_pattern));

#ifdef HAVE_HYPERSCAN
{
struct hs_list h = (struct hs_list)ndpi_malloc(sizeof(struct hs_list));
@@ -4293,6 +4290,10 @@ int ndpi_load_hostname_category(struct ndpi_detection_module_struct *ndpi_str,
return(-1);
}
#else

• AC_PATTERN_t ac_pattern;
• memset(&ac_pattern, 0, sizeof(ac_pattern));
• if(ndpi_str->custom_categories.hostnames_shadow.ac_automa == NULL)
  return(-1);

@@ -4313,8 +4314,8 @@ int ndpi_enable_loaded_categories(struct ndpi_detection_module_struct ndpi_str)
/ First add the nDPI known categories matches */
for(i=0; category_match[i].string_to_match != NULL; i++)
ndpi_load_hostname_category(ndpi_str,

•                           category_match[i].string_to_match,
  

•                           category_match[i].protocol_category);
  

•                            category_match[i].string_to_match,
  

•                            category_match[i].protocol_category);
  

#ifdef HAVE_HYPERSCAN
if(ndpi_str->custom_categories.num_to_load > 0) {
@@ -4342,6 +4343,11 @@ int ndpi_enable_loaded_categories(struct ndpi_detection_module_struct *ndpi_str)
head = head->next;
}

• if(i != ndpi_str->custom_categories.num_to_load){

•  ndpi_free(expressions);
  

•  return(-1);
  

• }
• free_hyperscan_memory(ndpi_str->custom_categories.hostnames);
  ndpi_str->custom_categories.hostnames = (struct hs*)ndpi_malloc(sizeof(struct hs));

@@ -4367,6 +4373,7 @@ int ndpi_enable_loaded_categories(struct ndpi_detection_module_struct *ndpi_str)
}

 ndpi_str->custom_categories.to_load = NULL;

• ndpi_str->custom_categories.num_to_load = 0;

  if(rc < 0) {
  ndpi_free(ndpi_str->custom_categories.hostnames);
  root@localhost:/home/vagrant/nDPI#

[lucaderi]

Fixed with the latest patches