fileinfo fails to parse attached PE files and uses a lot of memory

[s3rvac]

fileinfo fails to parse the attached PE files and uses a lot of memory.

Input

Run

/usr/bin/time -v retdec-fileinfo FILE

where FILE is:

• 7CE5BB5CA99B3570514AF03782545D41213A77A0F93D4AAC8269823A8D3A58EF
• CCE461B6EB23728BA3B8A97B9BE84C0FB9175DB31B9949E64144198AB3F702CE
• 4A2A008CF1AEE9BA49D8D1DAA22D8E868365ACE633823D464478239F27ED4F18

Output

...
Error: Failed to parse the input file (it is probably corrupted). Detected format is: PE.
Command exited with non-zero status 6
        ...
        Maximum resident set size (kbytes): 6307444

The amount of consumed memory depends on the input file. However, it is too high for all the attached files. For example, the 7CE5... file has 2 MB, but fileinfo requires ~6 GB of RAM to analyze it.

Expected output

Please, verify (1) whether the attached files are indeed corrupted, and (2) why does fileinfo require so much memory to analyze them.

Configuration

• Commit: 8bf8507 (current master)
• 64b Arch Linux, GCC 7.2.1, Debug build of RetDec
• Also fails on 64b Debian Jessie, GCC 4.9.2

[MerovingianByte]

~6GB is not so bad. Better than over 120GB hahaha

[s3rvac]

Here are outputs from valgrind --tool=massif (heap profiler) and massif-visualizer (visualization) for the 7CE5... sample. From them, we can see that fileinfo requires too much memory to read the import directory (well, at least for the 7CE5... sample).

What is interesting is that when I disabled reading of the import directory, not only the analysis succeeded but fileinfo required only 20 MB of memory to analyze the file:

$ /usr/bin/time -v retdec-fileinfo 7CE5BB5CA99B3570514AF03782545D41213A77A0F93D4AAC8269823A8D3A58EF
Input file               : 7CE5BB5CA99B3570514AF03782545D41213A77A0F93D4AAC8269823A8D3A58EF.dat
CRC32                    : 582e9d0a
MD5                      : af570bf9700d97793c78743ea6ae4b6c
SHA256                   : 7ce5bb5ca99b3570514af03782545d41213a77a0f93d4aac8269823a8d3a58ef
File format              : PE
File class               : 64-bit
File type                : Executable file
Architecture             : x86-64
Endianness               : Little endian
Image base address       : 0x140000000
Entry point address      : 0x1403390d6
Entry point offset       : 0x1046d6
Entry point section name : .vmp1
Entry point section index: 6
Bytes on entry point     : d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3d3
Detected tool            : VMProtect (packer), combined heuristic
Detected tool            : Microsoft Linker (11.0) (linker), combined heuristic
Rich header offset       : 0x80
Rich header key          : 0x1ea6bea3
Rich header signature    : 00c7a09e0000000200cdc6f10000000e00cfee660000000a00cfc6f10000008600cec6f1000000f600b977fc
                           00000008009c9d1b0000000300010000000000f900d3ee660000001b00c9ee660000000100ccee6600000001
Overlay offset           : 0x185400
Overlay size             : 0x7ac00
...
        Maximum resident set size (kbytes): 20776
...

[ladislav-zezula]

Fixed in commits 5b139f5dd8485e99a082ef9ab7757e87e369a712 and 92b69d0