Skip to content

[ELF] Parallelize --gc-sections mark phase #189321

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
MaskRay merged 3 commits into from
Apr 2, 2026
Merged

[ELF] Parallelize --gc-sections mark phase #189321

Changes from 1 commit
Commits
Show all changes
3 commits
Select commit Hold shift + click to select a range
0d85ffa
[ELF] Parallelize --gc-sections mark phase
MaskRay Mar 30, 2026
62d7242
rename processSectionEdges
MaskRay Apr 1, 2026
1c5a48f
rebase
MaskRay Apr 2, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
98 changes: 95 additions & 3 deletions lld/ELF/MarkLive.cpp
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -30,6 +30,7 @@
#include "lld/Common/Strings.h"
#include "llvm/ADT/DenseMapInfoVariant.h"
#include "llvm/ADT/STLExtras.h"
#include "llvm/Support/Parallel.h"
#include "llvm/Support/TimeProfiler.h"
#include <variant>
#include <vector>
Expand Down Expand Up @@ -66,6 +67,7 @@ template <class ELFT, bool TrackWhyLive> class MarkLive {
LiveReason reason);
void markSymbol(Symbol *sym, StringRef reason);
void mark();
void markParallel();

template <class RelTy>
void resolveReloc(InputSectionBase &sec, const RelTy &rel, bool fromFDE);
Expand Down Expand Up @@ -225,6 +227,51 @@ void MarkLive<ELFT, TrackWhyLive>::scanEhFrameSection(EhInputSection &eh) {
}
}

// Walk all GC edges from sec and call fn(target, offset) for each edge.
template <class ELFT, class Fn>
static void
forEachEdge(Ctx &ctx, InputSectionBase &sec,

@smithp35 smithp35 Mar 31, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This quite a long way away in the source file from its only caller, markParallel. Could it be moved closer?

With a name like forEachEdge I'd expect that all it does is traverse the edges, but doesn't have any other side-effects beyond calling fn, with fn doing the work.

In this case it looks like function is doing most of the actual work of marking things live, and the fn is mostlt a continuation (for want of a better word), with marking the section live via the partition a kind of side-effect of making sure a section isn't visited twice!

May I suggest something like

// Helper function for markParallel. Walk all GC edges from sec, marking everything that needs to be, live.
// Call fn(target section, offset) for each edge, which will mark the section live and handle further processing
// of edges from that section. 
processSectionEdges(...

@MaskRay MaskRay Apr 1, 2026

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the suggestion. Moved the function and renamed to processSectionEdges with the comment.

const DenseMap<StringRef, SmallVector<InputSectionBase *, 0>>
&cNamedSections,
Fn fn) {
auto resolveEdge = [&](const auto &rel) {
Symbol &sym = sec.file->getRelocTargetSym(rel);
if (!sym.used)
sym.used = true;
if (auto *d = dyn_cast<Defined>(&sym)) {
if (auto *relSec = dyn_cast_or_null<InputSectionBase>(d->section)) {
uint64_t offset = d->value;
if (d->isSection()) {
offset += getAddend<ELFT>(ctx, sec, rel);
if (auto *ms = dyn_cast<MergeInputSection>(relSec);
ms && offset >= ms->content().size())
return;
}
if (auto *ms = dyn_cast<MergeInputSection>(relSec))
ms->getSectionPiece(offset).live = true;
fn(relSec, offset);
}
return;
}
if (auto *ss = dyn_cast<SharedSymbol>(&sym))
if (!ss->isWeak())
cast<SharedFile>(ss->file)->isNeeded = true;
for (InputSectionBase *csec : cNamedSections.lookup(sym.getName()))
fn(csec, 0);
};
const RelsOrRelas<ELFT> rels = sec.template relsOrRelas<ELFT>();
for (const typename ELFT::Rel &rel : rels.rels)
resolveEdge(rel);
for (const typename ELFT::Rela &rel : rels.relas)
resolveEdge(rel);
for (const typename ELFT::Crel &rel : rels.crels)
resolveEdge(rel);
for (InputSectionBase *isec : sec.dependentSections)
fn(isec, 0);
if (sec.nextInSectionGroup)
fn(sec.nextInSectionGroup, 0);
}

// Some sections are used directly by the loader, so they should never be
// garbage-collected. This function returns true if a given section is such
// section.
Expand Down Expand Up @@ -464,7 +511,12 @@ void MarkLive<ELFT, TrackWhyLive>::run() {

template <class ELFT, bool TrackWhyLive>
void MarkLive<ELFT, TrackWhyLive>::mark() {
// Mark all reachable sections.
if constexpr (!TrackWhyLive) {
if (ctx.partitions.size() == 1) {
markParallel();
return;
}
}
while (!queue.empty()) {
InputSectionBase &sec = *queue.pop_back_val();

Expand All @@ -487,6 +539,46 @@ void MarkLive<ELFT, TrackWhyLive>::mark() {
}
}

// Parallel mark using level-synchronized BFS with depth-limited inline
// recursion. Each parallelFor iteration processes a subtree up to depth 3
// (DFS for cache locality), then queues deeper discoveries for the next level.
template <class ELFT, bool TrackWhyLive>
void MarkLive<ELFT, TrackWhyLive>::markParallel() {
const size_t numThreads = parallel::getThreadCount();
auto visit = [&](InputSection *sec, int depth,
SmallVector<InputSection *, 0> &localQueue,
auto &self) -> void {
forEachEdge<ELFT>(ctx, *sec, cNamedSections,
[&](InputSectionBase *target, uint64_t offset) {
auto &part = reinterpret_cast<std::atomic<uint8_t> &>(
target->partition);
// Optimistic load-then-exchange avoids expensive atomic
// RMW on already-visited sections.
if (part.load(std::memory_order_relaxed) != 0 ||
part.exchange(1, std::memory_order_relaxed) != 0)
return;
if (auto *s = dyn_cast<InputSection>(target)) {
if (depth < 3)
self(s, depth + 1, localQueue, self);
else
localQueue.push_back(s);
}
});
};

while (!queue.empty()) {
auto queues =
std::make_unique<SmallVector<InputSection *, 0>[]>(numThreads);
parallelFor(0, queue.size(), [&](size_t i) {
const unsigned tid = parallel::getThreadIndex();
visit(queue[i], 0, queues[tid], visit);
});
queue.clear();
for (size_t t = 0; t < numThreads; ++t)
queue.append(std::move(queues[t]));
}
}

// Move the sections for some symbols to the main partition, specifically ifuncs
// (because they can result in an IRELATIVE being added to the main partition's
// GOT, which means that the ifunc must be available when the main partition is
Expand Down Expand Up @@ -531,8 +623,8 @@ template <class ELFT> void elf::markLive(Ctx &ctx) {
return;
}

for (InputSectionBase *sec : ctx.inputSections)
sec->markDead();
parallelForEach(ctx.inputSections,
[](InputSectionBase *sec) { sec->markDead(); });

// Follow the graph to mark all live sections.
for (unsigned i = 1, e = ctx.partitions.size(); i <= e; ++i)
Expand Down