Some questions about DPC++ compilation phases.

[malixian]

In order to understand the compilation process of dpc++, I use the command clang -### -fsycl -fsycl-target=nvptx64-nvidia-cuda to print the stage of compilation. We can find that(my dpc++ version is not the latest, but the output of different versions of the compiler is basically the same):

1. "clang-13" "-cc1" "-triple" "nvptx64-nvidia-cuda-sycldevice" "-aux-triple" "x86_64-unknown-linux-gnu" "-fsycl-is-device" "-fdeclare-spirv-builtins" "-emit-llvm-bc" ..... "-mlink-builtin-bitcode" "/home/mlx/repos/llvm-cuda/build/lib/clang/13.0.0/../../clc/libspirv-nvptx64--nvidiacl.bc" "-mlink-builtin-bitcode" "/usr/local/cuda-11.5/nvvm/libdevice/libdevice.10.bc" "-o" "/tmp/vector_add-b7ec72.bc" "-x" "c++" "vector_add.cpp"
2. "/home/mlx/repos/llvm-cuda/build/bin/llvm-link" "/tmp/vector_add-b7ec72.bc" "-o" "/tmp/vector_add-6541d9.bc" "--suppress-warnings"
3. "/home/mlx/repos/llvm-cuda/build/bin/sycl-post-link" "-split=auto" "-ir-output-only" "-O2" "-spec-const=default" "-o" "/tmp/vector_add-ff4e38.bc" "/tmp/vector_add-6541d9.bc"
4. "/home/mlx/repos/llvm-cuda/build/bin/clang-13" "-cc1" "-triple" "nvptx64-nvidia-cuda-sycldevice" "-aux-triple" "x86_64-unknown-linux-gnu" "-fsycl-is-device" ..... "-S" .... "-o" "/tmp/vector_add-39b3c2.s" "-x" "ir" "/tmp/vector_add-ff4e38.bc"
5. "/usr/local/cuda-11.5/bin/ptxas" "-m64" "-O0" "--gpu-name" "sm_50" "--output-file" "/tmp/vector_add-11b29d.o" "/tmp/vector_add-39b3c2.s"
6. "/usr/local/cuda-11.5/bin/fatbinary" "-64" "--create" "/tmp/vector_add-145f0c.fatbin" "--image=profile=compute_50,file=/tmp/vector_add-39b3c2.s" "--image=profile=sm_50,file=/tmp/vector_add-11b29d.o"
7. "/home/mlx/repos/llvm-cuda/build/bin/clang-offload-wrapper" "-o=/tmp/wrapper-f7e361.bc" "-host=x86_64-unknown-linux-gnu" "-target=nvptx64" "-kind=sycl" "/tmp/vector_add-145f0c.fatbin"
8. "/home/mlx/repos/llvm-cuda/build/bin/llc" "-filetype=obj" "-o" "/tmp/vector_add-27168e.o" "/tmp/wrapper-f7e361.bc"
9. "/home/mlx/repos/llvm-cuda/build/bin/clang-13" "-cc1" "-triple" "x86_64-unknown-linux-gnu" ..... "-o" "/tmp/vector_add-c9a7c6.o" "-x" "c++" "vector_add.cpp"
10. "/usr/bin/ld" "-o" "vector_add.out" "/usr/lib/x86_64-linux-gnu/crt1.o" ..... "-lm" "-lgcc_s" "-lgcc" "-lsycl" "-lc" "-lgcc_s" "-lgcc" "/usr/lib/gcc/x86_64-linux-gnu/8/crtend.o" "/usr/lib/x86_64-linux-gnu/crtn.o"

My question is:

1. In th 4th phases, Why not use llc --march=nvptx64 .... instead of the very long clang command？ It seems to work after I tried it.
2. Why is llc used in the 8th step? I don't know what it is used for. Is it related to the 7th clang-offload-wrapper?

Thanks for your replay.

[Naghasan]

Hi,

Interesting questions,

1. The "upstream" way is to invoke clang rather than llc (I don't think there is a way to invoke llc via clang in upstream LLVM actually). In that mode, clang does act as llc (goes straight into invoking the optimizer and backend). I think from an llvm/clang point of view, this is simply because clang is self-contained so shipping llc is redundant, plus it would also add more code to maintain for a job already done by clang. When we did the driver, we followed the upstream "way" to reuse the existing compiler invocation logic (for instance you can reuse target specific flags out-of-the box) and ease upstreaming work . The HIP target follows the same pattern.

You can see this "upstream" way in action if you do something like clang -save-temps -### foo.cpp -o foo. The call to llc you are referring to will be a call to clang -cc1 and the following should be a clang -cc1as (calling LLVM's assembly).

2. That's part of the main SYCL integration, unlike the 4th phase, the action behind that invocation is coming from a shared logic between all backends. We just hooked the NVPTX path with this. In short, clang-offload-wrapper bundles the resulting binaries and manifest for the targets and store them into a specifically named data section (OpenMP offload uses the same approach). The result is an LLVM modules and the call to llc is just to format that module into an object file that can simply linked. I'm not sure why Intel did it that way rather than using the existing upstream paths. This is however done for a very specific job (create a data section in an object file). I think @kbobrovs could provide more details on the motivations.