Implement telfhash for ELF import table #936

HoundThe · 2021-03-20T00:23:45Z

I've added tlsh https://github.com/trendmicro/tlsh code into the project and used it to calculate telfhash for ELF import tables.

include/retdec/fileformat/utils/tlsh/tlsh_impl.h

PeterMatula · 2021-03-30T10:00:43Z

Lets run TC tests.

PeterMatula

linux-doxygen-build failed.
win-build failed - looks like it has trouble compiling TLSH. Solving errors on WIndows is a major pita. If you don't have a windows machine, and cannot get you hands on some virtualbox, let me know. We could probably give you access to a windows server, but it is kinda last resort - having it locally in virutalbox is better.
TLSH sources must not be between other RetDec sources. Move it to /deps/tlsh. Create CMake library target for it - look for example at tinyxml2 which should be similar. I can help with CMake if needed. Also, add at least license file to sources, and some version identification, e.g. version file if it exists - preferably use some tagged TLSH version, not random commit at master. Also, add TLSH entry to our /LICENSE-THIRD-PARTY file.
I also expect PR in retdec-regression-tests that on some real-world input sample tests both plain and json values. Place it in /tools/fileinfo/features.
Can you send me a file, where it produces some hash? I tried it on retdec-regression-tests/integration/ack/2018-09-13/ack.x64.gcc.O0.g.elf and I got newly added entries for SHA256, MD5, CRC32, but no TLSH. Is it working for you? M'I doing something wrong? This is exactly the thing that regression tests would solve automatically.
After you fix these issues, and test the implementation on those 450k samples I sent, change the PR from draft to a full PR, and let me know. I will go over it again and merge it.

include/retdec/fileformat/types/import_table/elf_import_table.h

include/retdec/fileformat/types/import_table/import_table.h

src/fileformat/types/import_table/elf_import_table.cpp

PeterMatula · 2021-03-30T11:47:46Z

src/fileformat/types/import_table/elf_import_table.cpp

+		imported_symbols.push_back(funcName);
+	}
+
+	std::sort(imported_symbols.begin(), imported_symbols.end());


If you are sorting, why not use std::set? Are duplicates possible/desirable? If yes, and we want to keep them, std::vector is ok. If we don't want them, std::set would solve it for us.
Does telfhash allow duplicates in this container?

Also a thing, is the sorting algorithm behaving the same as in the telfhash implementation? I think it is possible to somehow control the string sorting in C++? There might be differences. And even it there are not, a comment on this would be helpful so that no one screws it up later.

Original implementation seems to use a python list and not check for duplicates, so I think it allows duplicates. Python sort and C++ one on strings should sort lexicographically. But I'll be more sure after further testing.

I may be confusing C++ with Linux, but there are definitely more ways to sort strings - Linux sort, or maybe some other app, is controlled by LOCALE (I think). Just check iPython <-> C++, and add some note. In general, because we try to exactly reimplement some other code here, comments on these things are highly desirable.

src/fileformat/types/import_table/elf_import_table.cpp

src/fileformat/types/import_table/import_table.cpp

HoundThe · 2021-03-30T12:35:14Z

Can you send me a file, where it produces some hash? I tried it on retdec-regression-tests/integration/ack/2018-09-13/ack.x64.gcc.O0.g.elf and I got newly added entries for SHA256, MD5, CRC32, but no TLSH. Is it working for you? M'I doing something wrong? This is exactly the thing that regression tests would solve automatically.

This one for example https://www.virustotal.com/gui/file/fb5b5659801beccc790444f5fecb6acc74c10469acf7554f8f1eeea033bbbfe8
I think the file you tested it on has too few symbols to generate the hash from. I am not sure what is the exact minimum input, from the tlsh source it seems to be this #define MIN_DATA_LENGTH 50, the binary you've tried has data length of only 6, as the only non-filtred import symbol is printf. Can be tested by the original telfhash implementation, which outputs tnull:

$ telfhash retdec-regression-tests/integration/ack/2018-09-13/ack.x64.gcc.O0.g.elf -d
ELFCLASS64
8 symbols found
1 symbols considered:
printf
retdec-regression-tests/integration/ack/2018-09-13/ack.x64.gcc.O0.g.elf  tnull

PeterMatula · 2021-03-31T08:51:57Z

Lets run TC tests.

HoundThe · 2021-04-01T20:29:12Z

I've moved TLSH to deps and wrote a CMakeList for it inspired by existing ones, but it's the first time I used CMake so I am not sure if I did it correctly, especially with the cond_add which I've set if RETDEC_ENABLE_FILEFORMAT (in option.cmake) is set.

I've added their license file content into the LICENSE-THIRD-PARTY file. But I am not sure what should I put into the license file and version file that I should put among the source files. I've used the latest tagged release https://github.com/trendmicro/tlsh/releases/tag/4.2.1 . The version values are in the CMakeList

HoundThe · 2021-04-02T22:43:46Z

After testing out the implementation on 464527 ELF binaries I've found few things about telfhash that I didn't really account for:

telfhash hashes also exported symbols, as in symbols that have value.
telfhash cares only about sections, if the binary has symbols in dynamic segments, telfhash doesn't use them even tho RetDec could because it's capable to extract them.

The first one results in a situation where we calculated telfhash, but because there are no imported symbols, there is no import table in the output, so we don't output telfhash. Maybe use something else strictly for the import table and use telfhash separately? That could also give the ability to use symbols from the dynamic segments.

…phashes, create default imphash for ELF

HoundThe · 2021-04-07T18:32:33Z

I moved the telfhash to the ElfFormat and kept the existing implementation in import table as a general import hash. I've also added TLSH hash to every import table, because I thought why not, but I can remove it.

I've tested it and seems to work well compared to original implementation, I think it's ready for re-review.

include/retdec/fileformat/file_format/elf/elf_format.h

src/fileformat/types/import_table/import_table.cpp

Implement telfhash for import table and add TLSH to the project

4bb8979

HoundThe requested a review from PeterMatula March 20, 2021 00:28

s3rvac reviewed Mar 20, 2021

View reviewed changes

include/retdec/fileformat/utils/tlsh/tlsh_impl.h Outdated Show resolved Hide resolved

HoundThe added 2 commits March 23, 2021 13:11

comment the import symbol filter regexes

6505da0

Use std::set for faster lookup

2e5d12c

PeterMatula requested changes Mar 30, 2021

View reviewed changes

HoundThe added 2 commits March 30, 2021 14:46

Address code review comments

c27d3d7

better formatting

0ec716c

HoundThe added 2 commits April 1, 2021 20:26

Move TLSH to deps/ using cmake

71d31cc

Forgot to commit tlsh headers

6611b0f

HoundThe added 4 commits April 2, 2021 00:22

Restructure elf_format to get symbols in the same manner as telfhash

7d5382d

Ignore symbols from dynamic segments

53ac63e

First exclude then convert to lower_case

9fca602

mask out symbol visibility from others

3d1720d

HoundThe added 2 commits April 6, 2021 16:37

Move telfhash outside import table to elf_format, use TLSH for all im…

1209eb7

…phashes, create default imphash for ELF

Fix uninitialized value

7321cf0

HoundThe mentioned this pull request Apr 6, 2021

Add tests for telfhash feature avast/retdec-regression-tests#85

Merged

Fixed TLSH build on Windows

052432a

HoundThe marked this pull request as ready for review April 7, 2021 00:45

HoundThe requested a review from PeterMatula April 7, 2021 18:32

PeterMatula and others added 4 commits April 13, 2021 13:38

fileformat/CMakeLists.txt: do not add tlsh-related stuff

536bef1

deps/tlsh: refactor CMake

711463d

cmake/options.cmake: move TLSH to deps section

6299548

deps/tlsh/cmake: add new line at the end

10217f7

PeterMatula and others added 3 commits April 14, 2021 11:26

fileformat/elf_format: C comment -> C++ comment

9c175a3

fileformat/elf_import_table.h: add missing new line

6ddeb7e

fileformat: remove trailing spaces

c5c91b0

PeterMatula reviewed Apr 14, 2021

View reviewed changes

PeterMatula merged commit 0cdc9a1 into avast:master Apr 14, 2021

PeterMatula mentioned this pull request Apr 14, 2021

Fileformat does not generate imphash for ELF binaries #286

Closed

PeterMatula added a commit that referenced this pull request Apr 14, 2021

CHANGELOG.md: add entry for #286, #936

b207930

HoundThe deleted the telfhash branch April 14, 2021 18:17

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implement telfhash for ELF import table #936

Implement telfhash for ELF import table #936

HoundThe commented Mar 20, 2021

PeterMatula commented Mar 30, 2021

PeterMatula left a comment •

edited

Loading

PeterMatula Mar 30, 2021

PeterMatula Mar 30, 2021

HoundThe Mar 30, 2021

PeterMatula Mar 30, 2021

HoundThe commented Mar 30, 2021

PeterMatula commented Mar 31, 2021

HoundThe commented Apr 1, 2021 •

edited

Loading

HoundThe commented Apr 2, 2021 •

edited

Loading

HoundThe commented Apr 7, 2021

Implement telfhash for ELF import table #936

Implement telfhash for ELF import table #936

Conversation

HoundThe commented Mar 20, 2021

PeterMatula commented Mar 30, 2021

PeterMatula left a comment • edited Loading

Choose a reason for hiding this comment

PeterMatula Mar 30, 2021

Choose a reason for hiding this comment

PeterMatula Mar 30, 2021

Choose a reason for hiding this comment

HoundThe Mar 30, 2021

Choose a reason for hiding this comment

PeterMatula Mar 30, 2021

Choose a reason for hiding this comment

HoundThe commented Mar 30, 2021

PeterMatula commented Mar 31, 2021

HoundThe commented Apr 1, 2021 • edited Loading

HoundThe commented Apr 2, 2021 • edited Loading

HoundThe commented Apr 7, 2021

PeterMatula left a comment •

edited

Loading

HoundThe commented Apr 1, 2021 •

edited

Loading

HoundThe commented Apr 2, 2021 •

edited

Loading