Skip to content

Commit 5004c59

Browse files
bd1976brisbd1976bris
authored
[DTLTO][Clang] Add support for Integrated Distributed ThinLTO (#147265)
This patch introduces support for Integrated Distributed ThinLTO (DTLTO) in Clang. DTLTO enables the distribution of ThinLTO backend compilations via external distribution systems, such as Incredibuild, during the traditional link step: https://llvm.org/docs/DTLTO.html. Testing: - `lit` test coverage has been added to Clang's Driver tests. - The DTLTO cross-project tests will use this Clang support. For the design discussion of the DTLTO feature, see: #126654
1 parent 51b6f64 commit 5004c59

File tree

5 files changed

+117
-19
lines changed

5 files changed

+117
-19
lines changed

‎clang/docs/ThinLTO.rst‎

Lines changed: 32 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -240,6 +240,38 @@ The ``BOOTSTRAP_LLVM_ENABLE_LTO=Thin`` will enable ThinLTO for stage 2 and
240240
stage 3 in case the compiler used for stage 1 does not support the ThinLTO
241241
option.
242242

243+
Integrated Distributed ThinLTO (DTLTO)
244+
--------------------------------------
245+
246+
Integrated Distributed ThinLTO (DTLTO) enables the distribution of backend
247+
ThinLTO compilations via external distribution systems, such as Incredibuild,
248+
during the traditional link step.
249+
250+
The implementation is documented here: https://llvm.org/docs/DTLTO.html.
251+
252+
DTLTO requires the LLD linker (``-fuse-ld=lld``).
253+
254+
``-fthinlto-distributor=<path>``
255+
- Specifies the ``<path>`` to the distributor process executable for DTLTO.
256+
- If specified, ThinLTO backend compilations will be distributed by LLD.
257+
258+
``-Xthinlto-distributor=<arg>``
259+
- Passes ``<arg>`` to the distributor process (see ``-fthinlto-distributor=``).
260+
- Can be specified multiple times to pass multiple options.
261+
- Multiple options can also be specified by separating them with commas.
262+
263+
Examples:
264+
- ``clang -flto=thin -fthinlto-distributor=incredibuild.exe -Xthinlto-distributor=--verbose,--j10 -fuse-ld=lld``
265+
- ``clang -flto=thin -fthinlto-distributor=$(which python) -Xthinlto-distributor=incredibuild.py -fuse-ld=lld``
266+
267+
If ``-fthinlto-distributor=`` is specified, Clang supplies the path to a
268+
compiler to be executed remotely to perform the ThinLTO backend
269+
compilations. Currently, this is Clang itself.
270+
271+
Note that currently, DTLTO is only supported in some LLD flavors. Support can
272+
be added to other LLD flavours in the future.
273+
See `DTLTO <https://lld.llvm.org/dtlto.html>`_ for more information.
274+
243275
More Information
244276
================
245277

‎clang/include/clang/Driver/Options.td‎

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -990,6 +990,13 @@ def Xlinker : Separate<["-"], "Xlinker">, Flags<[LinkerInput, RenderAsInput]>,
990990
Visibility<[ClangOption, CLOption, FlangOption]>,
991991
HelpText<"Pass <arg> to the linker">, MetaVarName<"<arg>">,
992992
Group<Link_Group>;
993+
def Xthinlto_distributor_EQ : CommaJoined<["-"], "Xthinlto-distributor=">,
994+
Flags<[LinkOption]>,
995+
Visibility<[ClangOption, CLOption]>,
996+
HelpText<"Pass <arg> to the ThinLTO distributor process. Can be specified "
997+
"multiple times or with comma-separated values.">,
998+
MetaVarName<"<arg>">,
999+
Group<Link_Group>;
9931000
def Xoffload_linker : JoinedAndSeparate<["-"], "Xoffload-linker">,
9941001
Visibility<[ClangOption, FlangOption]>,
9951002
HelpText<"Pass <arg> to the offload linkers or the ones identified by -<triple>">,
@@ -4249,7 +4256,12 @@ def ffinite_loops: Flag<["-"], "ffinite-loops">, Group<f_Group>,
42494256
def fno_finite_loops: Flag<["-"], "fno-finite-loops">, Group<f_Group>,
42504257
HelpText<"Do not assume that any loop is finite.">,
42514258
Visibility<[ClangOption, CC1Option]>;
4252-
4259+
def fthinlto_distributor_EQ : Joined<["-"], "fthinlto-distributor=">,
4260+
Group<f_Group>,
4261+
HelpText<"Path to the ThinLTO distributor process. If specified, "
4262+
"ThinLTO backend compilations will be distributed by LLD">,
4263+
MetaVarName<"<path>">,
4264+
Visibility<[ClangOption, CLOption]>;
42534265
def ftrigraphs : Flag<["-"], "ftrigraphs">, Group<f_Group>,
42544266
HelpText<"Process trigraph sequences">, Visibility<[ClangOption, CC1Option]>;
42554267
def fno_trigraphs : Flag<["-"], "fno-trigraphs">, Group<f_Group>,

‎clang/lib/Driver/ToolChains/CommonArgs.cpp‎

Lines changed: 11 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -1320,6 +1320,17 @@ void tools::addLTOOptions(const ToolChain &ToolChain, const ArgList &Args,
13201320
if (Args.hasArg(options::OPT_ftime_report))
13211321
CmdArgs.push_back(
13221322
Args.MakeArgString(Twine(PluginOptPrefix) + "-time-passes"));
1323+
1324+
if (Arg *A = Args.getLastArg(options::OPT_fthinlto_distributor_EQ)) {
1325+
CmdArgs.push_back(
1326+
Args.MakeArgString("--thinlto-distributor=" + Twine(A->getValue())));
1327+
CmdArgs.push_back(
1328+
Args.MakeArgString("--thinlto-remote-compiler=" +
1329+
Twine(ToolChain.getDriver().getClangProgramPath())));
1330+
1331+
for (auto A : Args.getAllArgValues(options::OPT_Xthinlto_distributor_EQ))
1332+
CmdArgs.push_back(Args.MakeArgString("--thinlto-distributor-arg=" + A));
1333+
}
13231334
}
13241335

13251336
void tools::addOpenMPRuntimeLibraryPath(const ToolChain &TC,

‎clang/test/Driver/DTLTO/dtlto.c‎

Lines changed: 48 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,48 @@
1+
// REQUIRES: lld
2+
3+
/// Check DTLTO options are forwarded to the linker.
4+
5+
/// Check that options are forwarded as expected with --thinlto-distributor=.
6+
// RUN: %clang -flto=thin %s -### -fuse-ld=lld --target=x86_64-linux-gnu \
7+
// RUN: -Xthinlto-distributor=a1 -Xthinlto-distributor=a2,a3 \
8+
// RUN: -fthinlto-distributor=d.exe -Werror 2>&1 | \
9+
// RUN: FileCheck %s --check-prefix=FORWARD
10+
11+
// FORWARD: ld.lld
12+
// FORWARD-SAME: "--thinlto-distributor=d.exe"
13+
// FORWARD-SAME: "--thinlto-remote-compiler={{.*}}clang{{[^\"]*}}"
14+
// FORWARD-SAME: "--thinlto-distributor-arg=a1"
15+
// FORWARD-SAME: "--thinlto-distributor-arg=a2"
16+
// FORWARD-SAME: "--thinlto-distributor-arg=a3"
17+
18+
/// Check that options are not added without --thinlto-distributor= and
19+
/// that a warning is issued for unused -Xthinlto-distributor options.
20+
// RUN: %clang -flto=thin %s -### -fuse-ld=lld --target=x86_64-linux-gnu \
21+
// RUN: -Xthinlto-distributor=a1 -Xthinlto-distributor=a2,a3 2>&1 | \
22+
// RUN: FileCheck %s --check-prefix=NODIST --implicit-check-not=distributor \
23+
// RUN: --implicit-check-not=remote-compiler
24+
25+
// NODIST: warning: argument unused during compilation: '-Xthinlto-distributor=a1'
26+
// NODIST: warning: argument unused during compilation: '-Xthinlto-distributor=a2,a3'
27+
// NODIST: ld.lld
28+
29+
/// Check the expected arguments are forwarded by default with only
30+
/// --thinlto-distributor=.
31+
// RUN: %clang -flto=thin %s -### -fuse-ld=lld --target=x86_64-linux-gnu \
32+
// RUN: -fthinlto-distributor=d.exe -Werror 2>&1 | \
33+
// RUN: FileCheck %s --check-prefix=DEFAULT --implicit-check-not=distributor \
34+
// RUN: --implicit-check-not=remote-compiler
35+
36+
// DEFAULT: ld.lld
37+
// DEFAULT-SAME: "--thinlto-distributor=d.exe"
38+
// DEFAULT-SAME: "--thinlto-remote-compiler={{.*}}clang{{[^\"]*}}"
39+
40+
/// Check that nothing is forwarded when the compiler is not in LTO mode, and that
41+
/// appropriate unused option warnings are issued.
42+
// RUN: %clang %s -### -fuse-ld=lld --target=x86_64-linux-gnu \
43+
// RUN: -fthinlto-distributor=d.exe 2>&1 | \
44+
// RUN: FileCheck %s --check-prefix=NOFLTO --implicit-check-not=distributor \
45+
// RUN: --implicit-check-not=remote-compiler
46+
47+
// NOFLTO: warning: argument unused during compilation: '-fthinlto-distributor=d.exe'
48+
// NOFLTO: ld.lld

‎cross-project-tests/dtlto/ld-dtlto.c‎

Lines changed: 13 additions & 18 deletions
Original file line numberDiff line numberDiff line change
@@ -5,13 +5,11 @@
55

66
// RUN: rm -rf %t && mkdir %t && cd %t
77

8-
// RUN: %clang --target=x86_64-linux-gnu -c -flto=thin %s -o dtlto.o
9-
10-
// RUN: ld.lld dtlto.o \
11-
// RUN: --thinlto-distributor=%python \
12-
// RUN: --thinlto-distributor-arg=%llvm_src_root/utils/dtlto/local.py \
13-
// RUN: --thinlto-remote-compiler=%clang \
14-
// RUN: --thinlto-remote-compiler-arg=--save-temps
8+
// RUN: %clang --target=x86_64-linux-gnu %s -flto=thin -fuse-ld=lld \
9+
// RUN: -fthinlto-distributor=%python \
10+
// RUN: -Xthinlto-distributor=%llvm_src_root/utils/dtlto/local.py \
11+
// RUN: -Wl,--thinlto-remote-compiler-arg=--save-temps \
12+
// RUN: -nostdlib -Werror
1513

1614
/// Check that the required output files have been created.
1715
// RUN: ls | sort | FileCheck %s
@@ -22,18 +20,15 @@
2220
/// Linked ELF.
2321
// CHECK: {{^}}a.out{{$}}
2422

25-
/// Produced by the bitcode compilation.
26-
// CHECK-NEXT: {{^}}dtlto.o{{$}}
27-
2823
/// --save-temps output for the backend compilation.
29-
// CHECK-NEXT: {{^}}dtlto.s{{$}}
30-
// CHECK-NEXT: {{^}}dtlto.s.0.preopt.bc{{$}}
31-
// CHECK-NEXT: {{^}}dtlto.s.1.promote.bc{{$}}
32-
// CHECK-NEXT: {{^}}dtlto.s.2.internalize.bc{{$}}
33-
// CHECK-NEXT: {{^}}dtlto.s.3.import.bc{{$}}
34-
// CHECK-NEXT: {{^}}dtlto.s.4.opt.bc{{$}}
35-
// CHECK-NEXT: {{^}}dtlto.s.5.precodegen.bc{{$}}
36-
// CHECK-NEXT: {{^}}dtlto.s.resolution.txt{{$}}
24+
// CHECK-NEXT: {{^}}ld-dtlto-[[TMP:[a-zA-Z0-9_]+]].s{{$}}
25+
// CHECK-NEXT: {{^}}ld-dtlto-[[TMP]].s.0.preopt.bc{{$}}
26+
// CHECK-NEXT: {{^}}ld-dtlto-[[TMP]].s.1.promote.bc{{$}}
27+
// CHECK-NEXT: {{^}}ld-dtlto-[[TMP]].s.2.internalize.bc{{$}}
28+
// CHECK-NEXT: {{^}}ld-dtlto-[[TMP]].s.3.import.bc{{$}}
29+
// CHECK-NEXT: {{^}}ld-dtlto-[[TMP]].s.4.opt.bc{{$}}
30+
// CHECK-NEXT: {{^}}ld-dtlto-[[TMP]].s.5.precodegen.bc{{$}}
31+
// CHECK-NEXT: {{^}}ld-dtlto-[[TMP]].s.resolution.txt{{$}}
3732

3833
/// No files are expected after.
3934
// CHECK-NOT: {{.}}

0 commit comments

Comments
 (0)