LLVM build fails with "Target X86 is not in the set of libraries"

[testhound]

When I try to build I receive the following error:

Built target retdec-configtool
-- Could NOT find Python module pygments
-- Could NOT find Python module pygments.lexers.c_cpp
-- Found Python module yaml
-- LLVM host triple: x86_64-unknown-linux-gnu
-- LLVM default target triple: x86_64-unknown-linux-gnu
-- Building with -fPIC
-- Constructing LLVMBuild project information
-- Linker detection: GNU ld
-- Targeting X86
CMake Error at cmake/modules/LLVM-Config.cmake:175 (message):
Target X86 is not in the set of libraries.
Call Stack (most recent call first):
cmake/modules/LLVM-Config.cmake:312 (llvm_map_components_to_libnames)
cmake/modules/AddOCaml.cmake:69 (explicit_map_components_to_libraries)
bindings/ocaml/all_backends/CMakeLists.txt:1 (add_ocaml_library)

[xkubov]

Hi, just to be sure are you trying to build the latest RetDec master? As we cannot test it on your system more details on how you tried to build RetDec could come handy. Like, what options did you use with cmake?

From the output it seems that the fail occurred while building included LLVM, during the cmake phase specifically. I don't know whether there might be some clues, but CMake generates error log when error occurs, can you send the log?

There should be something like this at the end of the output:

See also "/home/xkubov/projects/llvm/build/CMakeFiles/CMakeError.log".

[testhound]

Yes I am trying to build master. The steps I followed:

1. git clone https://github.com/avast/retdec.git
2. mkdir build && cd build
3. cmake .. -DCMAKE_INSTALL_PREFIX="install path"
4. make

Link to CmakeError.log (which does not appear helpful)
https://gist.github.com/testhound/42c46bf566bc22e0f6b47e70546aa7d1

Link to CMakeOutput.log:

https://gist.github.com/testhound/c8be6fe5723af8b26c80e37a695c78b9

[xkubov]

I've tested the build on Ubuntu 18.04 and had no problem. What version of Ubuntu and cmake do you use?

Also can you confirm that you are unable to build the following project too?

$ git clone https://github.com/avast/llvm
$ cd llvm
$ mkdir build && cd build
$ cmake .. -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_INCLUDE_TESTS=OFF -DLLVM_INCLUDE_EXAMPLES=OFF -DLLVM_INCLUDE_TOOLS=OFF
$ make

[testhound]

I am running Ubuntu 18.04.1 with cmake 3.10.2.

The cmake line above (step 4) fails for me.

[testhound]

I upgraded my cmake to 3.16.4 and still receive the failure. Here is the cmake output:

https://gist.github.com/testhound/19501f66c743c16e6bf903bc8f1b1486

Note following instructions for the latest llvm works:

1. git clone https://github.com/llvm/llvm-project.git
2. cd llvm-project
3. mkdir build && cd build
4. cmake ../llvm -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_INCLUDE_TESTS=OFF -DLLVM_INCLUDE_EXAMPLES=OFF -DLLVM_INCLUDE_TOOLS=OFF

[testhound]

Also this succeeds for me with the latest llvm:

1. git clone https://github.com/llvm/llvm-project.git
2. cd llvm-project/llvm
3. mkdir build && cd build
4. cmake ../ -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_INCLUDE_TESTS=OFF -DLLVM_INCLUDE_EXAMPLES=OFF -DLLVM_INCLUDE_TOOLS=OFF

[s3rvac]

Thank you for the additional info @testhound. Our LLVM is modified so it does not build parts that are not necessary for RetDec. This speeds up the build. One of the modifications is most probably the reason why the build fails for you with

Target X86 is not in the set of libraries.

I was able to build our LLVM via the following Dockerfile:

FROM ubuntu:bionic

RUN apt-get -y update && \
    DEBIAN_FRONTEND=noninteractive apt-get install -y   \
    build-essential                                     \
    cmake                                               \
    python                                              \
    git

RUN git clone https://github.com/avast/llvm && \
    cd llvm && \
    mkdir build && \
    cd build && \
    cmake .. -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_INCLUDE_TESTS=OFF -DLLVM_INCLUDE_EXAMPLES=OFF -DLLVM_INCLUDE_TOOLS=OFF && \
    make -j$(nproc)

It uses Ubuntu 18.04, CMake 3.10.2, Python 2.7, and GCC 7.4.0, just like you do (based on the provided logs).

Before we start blindly changing the code, I would like to be able to reproduce the issue. I wonder what is so specific about your system that we are unable to reproduce the issue... 🤔

[testhound]

I don't know what is so specific about my system. I tried a second Ubuntu 18.04 system and cmake fails there as well.

I tried two local build today, using "-DLLVM_LOCAL_DIR=". I tried llvm 9.0 and llvm 8.x. I was able to get past the intital cmake error above but ran into compiler errors as the llvm code appears to have diverged.

[s3rvac]

I was able to determine the difference by comparing your cmake log with mine. The difference is that you have ocaml-findlib and libctypes-ocaml installed in your system, which triggers the build of OCaml bindings:

-- OCaml bindings enabled.

I am now able to reproduce the issue with the following Dockerfile:

FROM ubuntu:bionic

RUN apt-get -y update && \
    DEBIAN_FRONTEND=noninteractive apt-get install -y   \
    build-essential                                     \
    cmake                                               \
    python                                              \
    ocaml-findlib                                       \
    libctypes-ocaml                                     \
    git

RUN git clone https://github.com/avast/llvm && \
    cd llvm && \
    mkdir build && \
    cd build && \
    cmake .. -DLLVM_TARGETS_TO_BUILD=X86 -DLLVM_INCLUDE_TESTS=OFF -DLLVM_INCLUDE_EXAMPLES=OFF -DLLVM_INCLUDE_TOOLS=OFF

When we were modifying LLVM sources, we probably forgot about some file as we have never tried building OCaml bindings. Let me take a look and try to fix the issue.

[s3rvac]

In commit 02c33a8, I have disabled the build of bindings when building LLVM. Reasons:

1. We do not need them. This should speed up the build.
2. Our modified version of LLVM does not build correctly when building OCaml bindings. Since we want to build only the very necessary parts, disable the build of all bindings.

Could you please try building the current RetDec's master to see if you are able to finish the build?

[testhound]

s3rvac thank you that solved my problem!

[s3rvac]

Great! Happy to help.