Bin2llvmir: decoder refactoring

[PeterMatula]

The current decoder is a relic that was hacked to use capstone2llvmir when we got rid of the old semantic models. We need to refactor/rewrite it in order to improve the quality of decompilation.

After #115 is solved, refactor/rewrite decoder in a way where control-flow pass is embedded in the new decoder and hacks in the current decoder are not needed.

Notes:

• Do not add/subtract magic constants in code (mainly in dealing with ranges).
• Use references in statically linked code detection #113 is related to this issue.
• This should help with many problems across entire decompiler (e.g. Decompilation takes too much memory #13).
• I will add some more things to keep in mind to this issue, so that I don't have to create a bunch of decoder related issues.

[PeterMatula]

In Delphi samples, there are often data structures in code section. Do not decode them as code. Use them, if possible, to identify code.

42d43b.zip
In this sample:

• Function table at 0x403010 is stored to eax in the 3rd instruction on entry point.
• Some vtable stuctures from address 0x4020A8 stored to edx later on entry point.

[PeterMatula]

HV16_DebugM3

• Make sure enter instruction at 0x401000 is decoded ok.
• There are data at 0x4010cf (256 byte array), do not decode it as code.

[PeterMatula]

Check if we detect all the functions in Delphi binary: d7ab65f96a211ae3e822eae251d6f05ce8c05f910d9ec80fe7006cfca6b9c753.

[0xBEEEF]

A general question about the Delphi examples. As you can see in the IDR, a Delphi program still contains almost all required metadata somewhere in the program. IDR therefore also succeeds in restoring inhertitance and recognizing constructures and deconstructures. Would it be conceivable to use all this information here? Unfortunately, IDR is no longer being developed, and has already done a lot of preparatory work here. Maybe you can contact the author, maybe he provides some code snippets for the Delphi recognition. Then you don't have to worry about what has already been implemented in IDR.

[PeterMatula]

@0xBEEEF I don't know much about Delphi, I will have to look into it - how IDR works, what info binaries contain, if we can use it, etc.

[PeterMatula]

Decoder phase was completely rewritten. It is still not perfect, but the basic principles are OK now - complete control flow reconstruction during decoding. It will take much more fine-tuning to consistently get close to IDA quality.

However, it turns out that most problems with slow and memory consuming decompilations are caused by the currently used LLVM IR to BIR converted, not the decoding in bin2llvmir. So this is the next most important thing to solve (#211).