Merge pull request #3 from ntop/dev

update to latest

.gitignore

@@ -10,8 +10,13 @@
 *.la
 .libs
 .dirstamp
+.autotools
+.cproject
+.project
+.settings
 stamp-h1
 /configure
+/configure.ac
 /config.guess
 /config.h
 /config.h.in
@@ -41,5 +46,6 @@ stamp-h1
 /src/lib/.deps/
 /src/lib/protocols/.deps/
 /src/lib/third_party/src/.deps/
+/src/include/ndpi_define.h
 /test-driver
 /tests/Makefile

.travis.yml

@@ -40,3 +40,10 @@ after_script:
 #- lcov --remove coverage.info 'tests/*' 'packages/*' 'm4/*' 'doc/*' --output-file coverage.info
 #- lcov --list coverage.info # debug before upload
 #- coveralls-lcov --repo-token ${COVERALLS_TOKEN} coverage.info # uploads to coveralls
+
+notifications:
+  email:
+    recipients:
+      - [email protected]
+    on_success: never
+    on_failure: always

CHANGELOG.md

@@ -0,0 +1,116 @@
+# CHANGELOG
+
+
+#### nDPI 2.2 (December 2017)
+
+## Main New Features
+
+* Custom protocol categories to allow personalization of protocols-categories mappings
+* DHCP fingerprinting
+* HTTP User Agent discovery
+
+
+## New Supported Protocols and Services
+
+* ICQ (instant messaging client)
+* YouTube Upload
+* LISP
+* SoundCloud
+* Sony PlayStation
+* Nintendo (switch) gaming protocol
+
+
+## Improvements
+
+*  Windows 10 detection from UA and indentation
+*  Determine STUN flows that turn into RTP
+*  Fixes for iQIYI and 1kxun
+*  Android fingerprint
+*  Added DHCP class identifier support
+
+------------------------------------------------------------------------
+
+#### nDPI 2.0 (May 2017)
+
+## Main New Features
+
+* nDPI Wireshark plugin for Layer-7 protocol dissection. The plugin, available via an extcap interface, passes Wireshark the nDPI-detected protocols by adding an ethernet packet trailer that is then interpreted and displayed inside the Wireshark GUI. Readme: https://github.com/ntop/nDPI/blob/dev/wireshark/README.md
+
+
+## New Supported Protocols and Services
+
+* STARTTLS
+* IMAPS
+* DNScrypt
+* QUIC (Quick UDP Internet Connections)
+* AMQP (Advanced Message Queueing Protocol)
+* Ookla (SpeedTest)
+* BJNP
+* AFP (Apple Filing Protocol)
+* SMPP (Short Message Peer-to-Peer)
+* VNC
+* OpenVPN
+* OpenDNS
+* RX protocol (used by AFS)
+* CoAP and MQTT (IoT specific protocols)
+* Cloudflare
+* Office 365
+* OCS
+* MS Lync
+* Ubiquity AirControl 2
+* HEP (Extensible Encapsulation Protocol)
+* WhatsApp Voice vs WhatsApp (chat, no voice)
+* Viber	
+* Wechat
+* Github
+* Hotmail
+* Slack
+* Instagram
+* Snapchat
+* MPEG TS protocol
+* Twitch
+* KakaoTalk Voice and Chat
+* Meu
+* EAQ
+* iQIYI media service
+* Weibo
+* PPStream
+
+
+## Improvements
+
+* SSH client/server version dissection
+* Improved SSL dissection
+* SSL server certificate detection
+* Added double tagging 802.1Q in dissection of vlan-tagged packets
+* Improved netBIOS dissection
+* Improved Skype detection
+* Improved Netflix traffic detection
+* Improved HTTP subprotocol matching
+* Implemented DHCP host name extraction
+* Updated Facebook detection by ip server ranges
+* Updated Twitter networks	  
+* Improved Microsoft detection
+* Enhanced Google detection
+* Improved BT-uTP protocol dissection
+* Added detection of Cisco datalink layer (Cisco hDLC and Cisco SLARP)
+
+
+#### Older releases
+
+#### 2014-03-21
+* improved support for eDonkey/eMule/Kademlia
+* improved support for PPLive
+
+#### 2014-03-20
+* code optimizations
+* consistency improvements
+* added support for new applications: Pando Media Booster
+* improved support for Steam
+* added support for new web services: Wikipedia, MSN, Amazon, eBay, CNN
+
+#### 2014-03-19
+* added new protocols: FTP, code improvements
+
+#### 2014-03-17
+* added new protocols: SOCKSv4, SOCKSv5, RTMP

README.md

@@ -17,7 +17,7 @@ In order to compile this library do
 
 To run tests do additionally:
 
-- make check
+- cd tests; ./do.sh
 
 Please note that the pre-requisites for compilation include:
 - GNU tools (autogen, automake, autoconf, libtool)
@@ -29,7 +29,7 @@ The entire procedure of adding new protocols in detail:
 
 1. Add new protocol together with its unique ID to: src/include/ndpi_protocol_ids.h
 2. Create a new protocol in: src/lib/protocols/
-3. Variables to be kept for the duration of the entire flow (as state variables) needs to be placed in: /include/ndpi_structs.h in ndpi_flow_tcp_struct (for TCP only), ndpi_flow_udp_struct (for UDP only), or ndpi_flow_struct (for both).
+3. Variables to be kept for the duration of the entire flow (as state variables) need to be placed in: src/include/ndpi_typedefs.h in ndpi_flow_tcp_struct (for TCP only), ndpi_flow_udp_struct (for UDP only), or ndpi_flow_struct (for both).
 4. Add a new entry for the search function for the new protocol in: src/include/ndpi_protocols.h
 5. Choose (do not change anything) a selection bitmask from: src/include/ndpi_define.h
 6. Add a new entry in ndpi_set_protocol_detection_bitmask2 in: src/lib/ndpi_main.c
@@ -40,6 +40,13 @@ The entire procedure of adding new protocols in detail:
 11. make
 12. make check
 
+### How to use nDPI to Block Selected Traffic
+
+You can use nDPI to selectively block selected Internet traffic by embedding it onto an application (remember that nDPI us just a library). Both [ntopng](https://github.com/ntop/ntopng) and [nProbe cento](http://www.ntop.org/products/netflow/nprobe-cento/) can do this.
+
+### DISCLAIMER
+While we do our best to detect network protocols, we cannot guarantee that our software is error free and 100% accurate in protocol detection. Please make sure that you respect the privacy of users and you have proper authorization to listen, capture and inspect network traffic.
+
 ### Creating A Source File Tar Ball
 
 If you want to distribute a source tar file of nDPI do:

README.nDPI

@@ -1,9 +1,10 @@
-rerequisites for Compilation
+Prerequisites for Compilation
 -----------------------------
 
 Prerequisites
 - GNU autotools/libtool
 - libpcap or PF_RING (optional but recommended)
+- apt-get install libjson-c-dev (for JSON support)
 
 On Ubuntu/Debian
 - apt-get install build-essential

autogen.sh

@@ -1,13 +1,17 @@
 #!/bin/sh
 
+NDPI_MAJOR="2"
+NDPI_MINOR="3"
+NDPI_PATCH="0"
+NDPI_VERSION_SHORT="$NDPI_MAJOR.$NDPI_MINOR.$NDPI_PATCH"
 
-/bin/rm -f configure config.h config.h.in src/lib/Makefile.in
+rm -f configure config.h config.h.in src/lib/Makefile.in
 
-AUTOCONF=$(which autoconf)
-AUTOMAKE=$(which automake)
-LIBTOOL=$(which libtool)
-LIBTOOLIZE=$(which libtoolize)
-AUTORECONF=$(which autoreconf)
+AUTOCONF=$(command -v autoconf)
+AUTOMAKE=$(command -v automake)
+LIBTOOL=$(command -v libtool)
+LIBTOOLIZE=$(command -v libtoolize)
+AUTORECONF=$(command -v autoreconf)
 
 if test -z $AUTOCONF; then
     echo "autoconf is missing: please install it and try again"
@@ -29,5 +33,15 @@ if test -z $AUTORECONF; then
     exit
 fi
 
+cat configure.seed | sed \
+    -e "s/@NDPI_MAJOR@/$NDPI_MAJOR/g" \
+    -e "s/@NDPI_MINOR@/$NDPI_MINOR/g" \
+    -e "s/@NDPI_PATCH@/$NDPI_PATCH/g" \
+    -e "s/@NDPI_VERSION_SHORT@/$NDPI_VERSION_SHORT/g" \
+    > configure.ac
+
 autoreconf -ivf
+cat configure | sed "s/#define PACKAGE/#define NDPI_PACKAGE/g" | sed "s/#define VERSION/#define NDPI_VERSION/g"  > configure.tmp
+cat configure.tmp > configure
+
 ./configure $*

configure.ac → configure.seed

@@ -1,4 +1,4 @@
-AC_INIT([libndpi], [1.8.0])
+AC_INIT([libndpi], [@NDPI_VERSION_SHORT@])
 
 AC_CONFIG_MACRO_DIR([m4])
 
@@ -7,8 +7,17 @@ AM_INIT_AUTOMAKE([foreign subdir-objects])
 LT_INIT
 
 AC_PROG_CC
+AM_PROG_CC_C_O
 AX_PTHREAD
 
+NDPI_MAJOR="@NDPI_MAJOR@"
+NDPI_MINOR="@NDPI_MINOR@"
+NDPI_PATCH="@NDPI_PATCH@"
+
+AC_DEFINE_UNQUOTED(NDPI_MAJOR_RELEASE, "${NDPI_MAJOR}", [nDPI major release])
+AC_DEFINE_UNQUOTED(NDPI_MINOR_RELEASE, "${NDPI_MINOR}", [nDPI minor release])
+AC_DEFINE_UNQUOTED(NDPI_PATCH_LEVEL,   "${NDPI_PATCH}", [nDPI patch level])
+
 if test -d ".git"; then :
      GIT_TAG=`git log -1 --format=%h`
      GIT_DATE=`git log -1 --format=%cd`
@@ -17,8 +26,7 @@ if test -d ".git"; then :
      # 
      #
      GIT_NUM=`git log --pretty=oneline | wc -l | tr -d '[[:space:]]'`
-     GIT_BRANCH=`git rev-parse --abbrev-ref HEAD`
-     GIT_RELEASE="${PACKAGE_VERSION}-${GIT_BRANCH}-${GIT_NUM}-${GIT_TAG}"
+     GIT_RELEASE="${PACKAGE_VERSION}-${GIT_NUM}-${GIT_TAG}"
 else
      GIT_RELEASE="${PACKAGE_VERSION}"
      GIT_DATE=`date`
@@ -40,7 +48,18 @@ SHORT_MACHINE=`uname -m | cut -b1-3`
 if test $SHORT_MACHINE = "arm"; then
      LIBNUMA=""
 else
-     LIBNUMA="-lnuma"
+     AC_CHECK_LIB([numa], [numa_available], [LIBNUMA="-lnuma"])
+fi
+
+
+HS_LIB=
+HS_INC=
+AC_ARG_WITH(hyperscan,     [  --with-hyperscan     Enable Intel Hyperscan (if available)])
+
+if test "${with_hyperscan}" == "yes"; then :
+   AC_CHECK_LIB([hs], [hs_compile_multi], AC_DEFINE_UNQUOTED(HAVE_HYPERSCAN, 1, [Intel Hyperscan is present]))
+   HS_INC=`pkg-config --cflags libhs`
+   HS_LIB=`pkg-config --libs libhs`
 fi
 
 if test -f $PCAP_HOME/libpcap/libpcap.a; then :
@@ -66,6 +85,7 @@ else
    fi
 fi
 
+dnl> https://github.com/json-c/json-c
 AC_ARG_ENABLE([json-c],
     AS_HELP_STRING([--disable-json-c], [Disable json-c support]))
 
@@ -78,28 +98,36 @@ AS_IF([test "x$enable_json_c" != "xno"], [
               LDFLAGS="$LDFLAGS $(pkg-config --libs json-c)"
               AC_CHECK_LIB(json-c, json_object_new_object, AC_DEFINE_UNQUOTED(HAVE_JSON_C, 1, [The JSON-C library is present]))
              ],
-             [])
+             [
+		JSONC_HOME="$HOME/json-c"
+	          if test -d "$JSONC_HOME"; then :
+		        CFLAGS="$CFLAGS -I $JSONC_HOME"
+			LDFLAGS="$LDFLAGS $JSONC_HOME/.libs/libjson-c.a"
+    		        AC_MSG_RESULT([Found json-c in $JSONC_HOME])
+		 	AC_DEFINE_UNQUOTED(HAVE_JSON_C, 1, [The JSON-C library is present])
+		  fi
+	     ])
        ])
 
-OLD_LIBS=$LIBS
-LIBS="-L/opt/napatech3/lib $LIBS"
-AC_CHECK_LIB([ntapi],
-             [NT_Init],
-             [PCAP_LIB="$PCAP_LIB -L/opt/napatech3/lib -lntapi"],
-             [], [] )
-LIBS=$OLD_LIBS
-
+AC_ARG_ENABLE([debug-messages],
+    AS_HELP_STRING([--enable-debug-messages], [Define NDPI_ENABLE_DEBUG_MESSAGES=1]), [
+	AC_DEFINE(NDPI_ENABLE_DEBUG_MESSAGES, 1, [Enable ndpi_debug_messages]) ])
 
 AC_CHECK_LIB(pthread, pthread_setaffinity_np, AC_DEFINE_UNQUOTED(HAVE_PTHREAD_SETAFFINITY_NP, 1, [libc has pthread_setaffinity_np]))
 
-AC_CONFIG_FILES([Makefile src/lib/Makefile example/Makefile tests/Makefile libndpi.pc])
-AC_CONFIG_HEADERS(config.h)
+AC_CONFIG_FILES([Makefile src/lib/Makefile example/Makefile tests/Makefile libndpi.pc src/include/ndpi_define.h])
+AC_CONFIG_HEADERS(src/include/ndpi_config.h)
 AC_SUBST(GIT_RELEASE)
+AC_SUBST(NDPI_MAJOR)
+AC_SUBST(NDPI_MINOR)
+AC_SUBST(NDPI_PATCH)
 AC_SUBST(SVN_DATE)
 AC_SUBST(JSON_C_LIB)
 AC_SUBST(PCAP_INC)
 AC_SUBST(PCAP_LIB)
 AC_SUBST(DL_LIB)
+AC_SUBST(HS_LIB)
+AC_SUBST(HS_INC)
 AC_SUBST(HAVE_PTHREAD_SETAFFINITY_NP)
 
 AC_OUTPUT

example/MacOS/README.md

@@ -0,0 +1,18 @@
+# How to use？
+
+You need to first compile the nDPI library as usual:
+
+- ./autogen.sh
+- ./configure
+- make
+
+Then open the Xcode project and you are ready to go. The default behavior is to analyze an embeded pcap file `capture.pcap`. You can change the behavior by changing command line input in `ViewController.m` file.
+
+# What does the XCode project do?
+
+It's a dummy Mac App project with a **Run** button. It doesn't modify any nDPI code except that it renamed the `main` function to `orginal_main` in `ndpiReader.c` (because the Mac App has it's own main function) and call the `orginal_main` with synthetic command line input from `ViewController.m` file when the **Run** button is clicked. 
+
+It also fixes some problems when compiling with Xcode. Some are listed below:
+- Add missed `NDPI_LOG_DEBUG2` macro definition implementation (defined as `NDPI_LOG_DEBUG2_XCODE_PROJ` in `ViewController.m`)
+- Add an empty ndpi_utils.h file to make `protocols/attic/ftp.c` and `protocols/attic/secondlife.c` can compile
+- Specially treat `ndpi_patricia.c` by not adding it into compilation source, since it's directly included in `ndpi_main.c`