ndpiSimpleIntegration can not analysis mysql

[INT2ECALL]

ndpiReader can tell me the netflow is mysql ,but why ndpiSimpleIntegration can not

[IvanNardi]

Could you attach a trace triggering the issue, please?

[INT2ECALL]

Could you attach a trace triggering the issue, please?

root@ubuntu:/home/ubuntu/Documents/nDPI-4.4-stable/example# ./ndpiReader -i ens33
.......
Detected protocols:
        Unknown              packets: 143           bytes: 10431         flows: 73
        DNS                  packets: 6             bytes: 640           flows: 3
        MDNS                 packets: 144           bytes: 66536         flows: 22
        NetBIOS              packets: 16            bytes: 1472          flows: 4
        SSDP                 packets: 156           bytes: 30760         flows: 26
        SMBv1                packets: 3             bytes: 702           flows: 3
        MySQL                packets: 25            bytes: 3248          flows: 1
        BitTorrent           packets: 18            bytes: 3420          flows: 4
        IGMP                 packets: 26            bytes: 1564          flows: 3
        TLS                  packets: 926           bytes: 178267        flows: 27
        SSH                  packets: 4             bytes: 372           flows: 1
        ICMPV6               packets: 111           bytes: 9690          flows: 10
        DHCPV6               packets: 16            bytes: 2252          flows: 12
        WSD                  packets: 33            bytes: 23262         flows: 4
        LLMNR                packets: 14            bytes: 1190          flows: 12
        Cloudflare           packets: 4             bytes: 252           flows: 1
        Azure                packets: 26            bytes: 5721          flows: 9
        Tencent              packets: 4             bytes: 264           flows: 1

this will tell me mysql.
but when i use ndpiSimpleIntegration 

root@ubuntu:/home/ubuntu/Documents/nDPI-4.4-stable/example# ./ndpiSimpleIntegration
usage: ./ndpiSimpleIntegration [PCAP-FILE-OR-INTERFACE]
.......
[      19, 2,    5][DETECTED] protocol: Unknown | app protocol: MDNS | category: Network
[      20, 2,    6] new flow
[      20, 2,    6][DETECTED] protocol: Unknown | app protocol: MDNS | category: Network
[      26, 2,    7] new flow
[      26, 2,    7][DETECTED] protocol: Unknown | app protocol: SSDP | category: System
[      34, 0,    8] new flow
[      34, 0,    8][DETECTED] protocol: Unknown | app protocol: SSDP | category: System
[      37, 0,    9] new flow
[      43, 0,   10] new flow
[      43, 0,   10][DETECTED] protocol: Unknown | app protocol: SSDP | category: System
[      61, 2,   11] new midstream-flow
[      62, 2,   11][DETECTED] protocol: Unknown | app protocol: TLS | category: Web

can not get mysql protocol

[IvanNardi]

Comparing results with real time traffic is a bit tricky. Could you reproduce the issue with a pcap? If so, could you share it, please?

[INT2ECALL]

Comparing results with real time traffic is a bit tricky. Could you reproduce the issue with a pcap? If so, could you share it, please?

thank for your reply , just use the ndpiSimpleIntegration ,and use command

mysql -u root -h 192.168.20.108 -p -P 3306 

[IvanNardi]

Even if it is a very simple application aiming only to show how to use nDPI API, ndpiSimpleIntegration seems to work with mysql flows, as expected:

ivan@ivan-Latitude-E6540:~/svnrepos/nDPI/example(dev)$ ./ndpiSimpleIntegration ../tests/pcap/mysql-8.pcap 
usage: ./ndpiSimpleIntegration [PCAP-FILE-OR-INTERFACE]
----------------------------------
nDPI version: 4.7.0-4071-4d11941d
 API version: 7696
libgcrypt...: 1.8.6internal
----------------------------------
Starting Thread 0
Starting Thread 1
Starting Thread 2
Starting Thread 3
[       1, 0,    0] new flow
[       4, 0,    0][DETECTED] protocol: Unknown | app protocol: MySQL | category: Database

To try to help you, you need to provide a trace triggering the issue (I don't have access to any mysql database)

[utoni]

There might be an issue regarding the protocol dissector. It should be able to detect the MySQL protocol, but not as app protocol. I'll try to replicate that when I am back home.

[INT2ECALL]

pcap is ok，but connect to interface is not ok

[INT2ECALL]

There might be an issue regarding the protocol dissector. It should be able to detect the MySQL protocol, but not as app protocol. I'll try to replicate that when I am back home.

that mean using ndpiSimpleIntegration and listen to my network interface cannot capture mysql i dont know why

[IvanNardi]

It is quite unlikely that ndpiSimpleIntegration provides different classifications depending on trace vs live traffic, especially with a trivial protocol like MySQL.
I am asking again: try to capture a pcap. Steps:

• start wireshark and start capturing traffic
• start ndpiSimpleIntegration with live traffic
• start mysql traffic

in THAT order.

This way you can capture a pcap while using ndpiSimpleIntegration with live traffic.

If I should guess, I think you are starting ndpiSimpleIntegration with live traffic after you started Mysql traffic...

[INT2ECALL]

https://tempsend.com/dbfbf/0d9e98/1680498646/%E5%B1%8F%E5%B9%95%E6%88%AA%E5%9B%BE+2023-04-03+130816.png

[INT2ECALL]

https://tempsend.com/gsfwx/0074d1/1680498717/123.pcapng

[INT2ECALL]

@IvanNardi i have upload pcap and img

[IvanNardi]

@INT2ECALL , thanks for the pcap; is has been very useful!
There are two different issues here:

1. [most important]: the mysql flows are not properly classified (by the library) -> we need to improve mysql detection
2. [minor issue]: ndpiSimpleIntegration doesn't classify any flows by port

In your case, you saw mysql results in ndpiReader and not in ndpiSimpleIntegration only because the classification by port works in the former and not in the latter (for any protocols). But the classification should be "by-dpi" (in both cases, obviously): that's the real issue.

Note to @utoni: if we want to fix 2), ndpi_detection_giveup should be called for every flows when it ends (without a proper classification) not only when they reach 0xFE packets (see around line 864), AFAIU

[IvanNardi]

@INT2ECALL , could I add part of your pcap to the public nDPI tests?

[INT2ECALL]

@IvanNardi yes ,and i will use ndpi to my production,

[IvanNardi]

@INT2ECALL , could you check if #1928 works for you, please?

[INT2ECALL]

@INT2ECALL , could you check if #1928 works for you, please?

i will do it later,thanks

[IvanNardi]

@INT2ECALL , if you want to compare ndpiReader and ndpiSimpleIntegration results, you might try ndpiReader -d, i.e. disabling classification by-port in ndpiReader. That should lead to the same results (famous last words...)

[INT2ECALL]

@IvanNardi another package ,ndpi can not detect ,wireshark can do

https://tempsend.com/whgnn

[INT2ECALL]

the mysql signature is in send request not response ,maybe this will help ndpi be better

[IvanNardi]

I think you are talking about "tcp.stream eq 1" flow.
I woudn't say that Wireshark is able to recognize it as a Mysql:

• the first packet (65) seems to be completely mangled: all the TCP payload is 0x73 and, even more strange, even some TCP options are set to 0x73. That is definitely not a Mysql packet.
• the second packet (67) seems to be a Mysql one, but I am not able to find any reference to command 133
• the 2 following packets are a TLS handshake (with a certificate with a name with "Mysql" string in it)

Please, note that Wireshark try to decode it as Mysql only because it uses the default port 3306, but it is not able to do it.

Bottom line: this flow MIGHT be some kind of TLS authentication of a Mysql session (but packet 65 is completely unexpected) but I don't know how to properly decode it

[INT2ECALL]

@IvanNardi 65 flow which i edited ,because i want to know mysql in request package not response package ,so i change it to check whether ndpi detect it from request or response

after i read mysql.c .it check from reponse from mysql server

this is important for me

Please, note that Wireshark try to decode it as Mysql only because it uses the default port 3306, but it is not able to do it.

so that mean it is not a good idea to detect mysql from 67 flow ,because it will make false positives

[IvanNardi]

As you noticed, detection is performed using (only) the first packet sent by the server.
We could improve it, to try detecting it asymmetrically, i.e. having only the packets in one direction, i.e. using the first packet sent by the client (without ANY packets from the server)
Just to be clear, nDPI doens't use the port to detect mysql, Wireshark does

[INT2ECALL]

@IvanNardi thank ! You are very handsome , this really help for me

[IvanNardi]

Create #1942 to keep track of Mysql asymmetric detection