black screen after high ram usage

[MerovingianByte]

When I tried decompiling an executable file, my ram usage sky rocket up to 98% or something, my computer heated and I got a nice black screen. The ram usage didn't go up immediately, but I started to notice significant slow down after some time, and I checked it was some process related to retdec that was the cause. Then just a black screen. Had to force shut down. Happened on windows 8.1. I know this information isn't particularly helpful, but try to decompile a fairly large 32-bit executable and watch the ram usage. I may be the only one who got a black screen, but can't be the only one with RAM usage sky rocketing to unreasonable levels.

As a side note, congratulations on the project. Even though I couldn't use this yet, I read https://retdec.com/web/files/publications/retdec-slides-botconf-2017.pdf and thought it's awesome. This program seems to decompile with the same quality or even better than hex-rays, and that's really something considering that all other decompilers I've tested so far besides hex-rays sucked. This decompiler seems very powerful. I guess I was just unlucky, but hopefully I'll be able to use it soon. Congratulations.

[mewmew]

Related to #13 and #16.

[s3rvac]

Unfortunately, decompilations of larger binaries (1 MB or more) may require a lot of RAM. When running decompilations, we advise to limit the maximal virtual memory before running decompile.sh to prevent swapping and unresponsiveness. On Linux, you can run e.g. ulimit -Sv 9863168 in your shell to limit the maximal virtual memory of processes that will be started from that shell (9863168 is 8 GB -- you will need to adjust the value based on your system). We will add this warning into the README.

Can you please share with us the input binary file that you tried to decompile?

[MerovingianByte]

I can't share the input binary file. But I will try to limit the maximum virtual memory. I have really high hopes for this project! Hope these memory issues can be fixed soon, keep up the awesome work!

[MerovingianByte]

I would just like to point out that even using selective decompilation "decompile.sh FILE --select-ranges 0x404000-0x405000 --select-decode-only" still resulted in black screen, and the range is pretty small, size 8A000(the whole file has range size 1A11000). I was trying to decompile the .bind section, which is where the program seem to start according to IDA. The program is probably packed. RETDEC even says that it's packed with PE diminisher v0.1. Don't know if that has something to do with anything, but just thought I should mention.

EDIT
Does retdec only work if the binary is not packed? It'd be very useful if it could be used to figure out packing schemes.

[s3rvac]

Does retdec only work if the binary is not packed? It'd be very useful if it could be used to figure out packing schemes.

It can (usually) detect if a binary file is packed. However, RetDec is currently able to unpack only binaries packed with UPX and MPRESS.

RETDEC even says that it's packed with PE diminisher v0.1. Don't know if that has something to do with anything, but just thought I should mention.

Yes, when the input file is packed and cannot be automatically unpacked (see above), then trying to decompile the file makes little sense as RetDec will be unable to decompile anything meaningful (maybe apart from the unpacking routine at the entry point in case of some packers).

[MerovingianByte]

Yes, when the input file is packed and cannot be automatically unpacked (see above), then trying to decompile the file makes little sense as RetDec will be unable to decompile anything meaningful (maybe apart from the unpacking routine at the entry point in case of some packers).

Well I'll be damned. In addition to the packing scheme, there seems to be protection against debugging. I can not run the PE with IDA or x32dbg by starting the process, I can only attach to the PE's process. And even then, the program crashes when it hits any break-point. Do you think usually even the protection is packed? I'd need to unpack it first, to then attack the protections? Do you think decompiling can help me with that? Thanks.

[MerovingianByte]

Just an update. I decrypted the PE and tried to decompile again. This time it went through many phases but still ran out of memory, after Global to local optimization. However, I did not get a black screen.

[MerovingianByte]

No point on leaving this open since there are 2 open related issues and I can't share the executable file for analysis. In other words this issue doesn't help other than saying retdec has memory problems.