Feature: Integration of Malheur

[jhg]

https://github.com/rieck/malheur of @rieck as in cuckoo modification of @brad-accuvant https://github.com/brad-accuvant/cuckoo-modified

[rieck]

@jhg @brad-accuvant Thanks for the effort! Let me know, if you run into problems.

[jhg]

@rieck I show that last commit in https://github.com/rieck/malheur is on Mar-2014, is malheur in active development? Has support still, bug fix, new features, optimization, or is abandoned development? I'm interesting in this because I like malheur, I like use machine learning for detect malware and I think is a good tool for a malware analysis system as Cuckoo, but also I think is relevant that development is active.

I think is possible with cherry-pick get commits of @brad-accuvant to a branch based on master of Cuckoo and check that all well for pull request it to Cuckoo master.

@jbremer or @botherder what about this for a new feature of Cuckoo?

[jbremer]

I was discussing earlier, even before Brad integrated Malheur, with Hugo Gascon to do machine learning based on Cuckoo. We have it on the roadmap for around the summer period. Also, malheur is, as you say, getting outdated and is not being updated anymore.

[rieck]

Malheur is not under active development any more—aside from bug fixes. This, however, might change, if new research questions pop up. Is there any other open-source project focusing on malware clustering?

[jhg]

@rieck I don't understand "questions pop up" (I'm spanish speaker and I only know 'pop up' as windows that is open from a web, for example for ads).

@jbremer exist a issue for speak about machine learning in Cuckoo? Is so good that it's was in the roadmap. I don't know status of machine learning in Cuckoo, if use malheur or not I don't know, but Cuckoo need machine learning.

[rieck]

@jhg There has been a lot of research on malware clustering. We might start working on Malheur again, if there are some new problems or ideas. One never knows. ;)

[jhg]

@rieck so good, and I like it, only that I don't know status of machine learning in Cuckoo it's must be coordinate with a member of Cuckoo team (as @jbremer or @botherder or @jekil for example). A idea for malheur is in the line of JohnTheRipper (https://github.com/magnumripper/JohnTheRipper) I see malheur is 90% C, I see can use OpenMP, then also check "neck of bottle" for translate to assembly the slow functions is a idea. I see that OpenMP is in readme as experimental, more development and test for do this stable is a good idea. Also is good for users make packages (deb, rpm, etc) not only compilation and installation, and for python modules send it to pip.

I only can proposal malheur as start point for don't start from scratch @rieck the rest is coordination of Cuckoo comunity and Cuckoo team. :D

[botherder]

Seems like this conversation stalled. Is the idea abandoned due to Malheur being inactive?

[rieck]

There are currently some ideas and I'll have a look at them the next weeks. See here: rieck/malheur#12 So the discussion is not dead but progressing slowly.

@jhg Providing binary packages or a Python interface would be really nice additions to Malheur. If you have some free time, I am happy to support you building these.

[doomedraven]

i will help with this :)

[doomedraven]

damn i forgot about it, i had backported it to v2 and it worked fine, @jbremer do you want PR of malheur?

[doomedraven]

but there a good discussion about it in brad's fork, i would ignore it, as after thousands of samples it so damn slow

[Mato-Z]

Hello, is any progress with integration of Malheur? Although Malheur development was ended, the primary idea - use of clustering for caught malware is still applicable. Would be possible use solution from https://github.com/brad-accuvant/cuckoo-modified and only modify it for current logs structure? If I understand the way which is described here https://blog.malwarebytes.com/threat-analysis/2015/08/automatic-analysis-using-malheur-and-cuckoo/ it would be the main method of processing Cuckoo outputs...

[doomedraven]

it easy to integrate/port to cuckoo upstream, but after many sampels as i told in last comment, cuckoo become so slow so i doubt what it will be integrated by devs, but they can respond better, you can easily integrate that for you if you really need it

[Mato-Z]

@doomedraven thanks for reply. So https://github.com/brad-accuvant/cuckoo-modified can process current Cuckoo log format on MIST?

[doomedraven]

no that mod fork is dead, and can't process current cuckoo results

install malheur like

cd /tmp/
git clone https://github.com/rieck/malheur.git
cd malheur
./bootstrap
./configure --prefix=/usr
make
sudo checkinstall
dpkg -i malheur_0.6.0-1_amd64.deb

just copy the malheur module to new cuckoo and adjust what is needed
add webtemplate which you can backport and adjust also
done

[Mato-Z]

@doomedraven many thanks. I will try it.