Add a used attribute

[mahkoh]

Add a used attribute to prevent symbols from being discarded.

[arcnmx]

Note that rust-lang/rust#30830 fixes this, and I consider the current behaviour to be a bug.

However, there's a bit of an unresolved question around whether the type should need to be marked pub or not.

[mahkoh]

That issue is somewhat related but doesn't fix this.

1. Your patch talks about LTO. LTO is irrelevant if there is only one crate.
2. Your patch only deals with no_mangle items. A used attribute can be applied to mangled items.
3. Your patch only deals with public items. A used attribute can be applied to private items.

[mahkoh]

I also don't agree with the patch that was merged. Items should be aggressively removed if they are not used. I'm not sure what your patch does exactly, but if it allows completely unused items to be present in the executable, then it should be reverted.

[arcnmx]

Erm I was a bit confused, sorry, let's make a graph!

#![feature(linkage)]
#![allow(dead_code)]

#[link_section = ".init_array"] #[linkage = "external"]
static F_STATIC_EXT: extern fn() = f_static_external;

#[link_section = ".init_array"] #[linkage = "external"]
pub static F_PUB_STATIC_EXT: extern fn() = f_pub_static_external;

#[link_section = ".init_array"]
static F_STATIC: extern fn() = f_static;

#[link_section = ".init_array"]
pub static F_PUB_STATIC: extern fn() = f_pub_static;

extern fn f_static() {
    println!("static");
}

extern fn f_static_external() {
    println!("static external");
}

extern fn f_pub_static_external() {
    println!("pub static external");
}

extern fn f_pub_static() {
    println!("pub static");
}

fn main() { }

rustc	rustc -O	rustc -O -Clto
static external	static external
pub static external	pub static external
static
pub static

#[linkage = "external"] is essentially #[used] and LTO needs to be adjusted to not kill those symbols.

[mahkoh]

#[linkage = "external"] is essentially #[used]

I don't think it is. Otherwise LLVM wouldn't have a separate mechanism for #[used].

[mahkoh]

Linkage has really nothing to do with this RFC.

[arcnmx]

Ah, fair, I wasn't aware llvm had a construct for that!

[nikomatsakis]

This seems like a reasonable thing to me, modulo bikeshedding the name. I feel like the name should be less generic, perhaps include something about "link" in it :)

[mahkoh]

@nikomatsakis This is the name used by gcc/clang and llvm for this feature. I don't know what else you'd call it.

[alexcrichton]

@arcnmx

Yes rust-lang/rust#30830 is different than this RFC as that largely just has to do with LTO. As @mahkoh indicated it is indeed the case that LLVM's llvm.used support is different than #[linkage = "external"]. I believe that this primarily has to do with the visibility in the symbol table itself.

For example, a #[used] global (as defined by this RFC) will simply not be omitted, but it will still follow the same linkage rules of before. Using the trick in rust-lang/rust#30817:

#![feature(linkage)]

#[export_name = "llvm.used"]
#[linkage = "appending"]
static mut _FOO: [*const u8; 1] = [
    &FOO_ADDR as *const _ as *const _
];

static FOO_ADDR: fn() = foo;

fn foo() {}

fn main() { }

If you take a look at the generated object file:

$ rustc foo.rs --emit obj
$ nm -g foo.o
0000000000000000 T main
                 U _ZN2rt10lang_start20h81372bffc43d2081zkyE
$ nm -a foo.o
0000000000000000 a foo.0.rs
0000000000000000 T main
0000000000000000 t .text.main
0000000000000000 t .text._ZN3foo20hea4f1d23bdee9020vaaE
0000000000000000 t .text._ZN4main20h7bc0c385e27c0783yaaE
                 U _ZN2rt10lang_start20h81372bffc43d2081zkyE
0000000000000000 t _ZN3foo20hea4f1d23bdee9020vaaE
0000000000000000 t _ZN4main20h7bc0c385e27c0783yaaE
0000000000000000 d _ZN8FOO_ADDR20hc3c6a3fb783d8c08saaE

Here we can see that foo and FOO_ADDR are in the object file, they're just not public. If they were marked with #[linkage = "external"] then they would indeed be public.

Another difference, I believe, is that LLVM has different optimization heuristics (such as inlining) for internal functions rather than external ones.

---

In my testing, however, I was surprised that this may not be as useful as originally appearing. One might expect that if the above program is compiled into a binary that it would indeed have the relevant symbols in the binary (but as private), it in fact does not. The reason for this is that we still pass --gc-sections to the linker, and the linker does indeed remove the symbols:

(note dynamic linkage is used here to reduce the number of stripped symbols

$ rustc -C prefer-dynamic -O foo.rs -C link-args=-Wl,--print-gc-sections
INFO:rustc_trans::back::link: LinkMeta { crate_name: "foo", crate_hash: Svh { hash: "d9f16434537bb629" } }
INFO:rustc_trans::back::link: preparing dylib? (false) from ["foo.0.o"] to "foo"
INFO:rustc_trans::back::linker: linking with ld.gold
INFO:rustc_trans::back::link: "cc" "-Wl,--as-needed" "-m64" "-L" "/home/alex/.multirust/toolchains/nightly/lib/rustlib/x86_64-unknown-linux-gnu/lib" "foo.0.o" "-o" "foo" "-Wl,--gc-sections" "-pie" "-Wl,-O1" "-nodefaultlibs" "-L" "/home/alex/.multirust/toolchains/nightly/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "-L" "/home/alex/.multirust/toolchains/nightly/lib/rustlib/x86_64-unknown-linux-gnu/lib" "-l" "std-17a8ccbd" "-l" "dl" "-l" "pthread" "-l" "gcc_s" "-l" "pthread" "-l" "c" "-l" "m" "-l" "rt" "-fuse-ld=gold" "-Wl,--print-gc-sections" "-l" "compiler-rt"
INFO:rustc_trans::back::link: linker stderr:
/usr/bin/ld.gold: removing unused section from '.rodata.cst4' in file '/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/Scrt1.o'
/usr/bin/ld.gold: removing unused section from '.data' in file '/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/Scrt1.o'
/usr/bin/ld.gold: removing unused section from '.bss' in file '/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/Scrt1.o'
/usr/bin/ld.gold: removing unused section from '.text' in file '/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o'
/usr/bin/ld.gold: removing unused section from '.data' in file '/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o'
/usr/bin/ld.gold: removing unused section from '.bss' in file '/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crti.o'
/usr/bin/ld.gold: removing unused section from '.data' in file '/usr/lib/gcc/x86_64-linux-gnu/4.8/crtbeginS.o'
/usr/bin/ld.gold: removing unused section from '.text' in file 'foo.0.o'
/usr/bin/ld.gold: removing unused section from '.data' in file 'foo.0.o'
/usr/bin/ld.gold: removing unused section from '.bss' in file 'foo.0.o'
/usr/bin/ld.gold: removing unused section from '.text._ZN3foo20hea4f1d23bdee9020vaaE' in file 'foo.0.o'
/usr/bin/ld.gold: removing unused section from '.data.rel.ro.local._ZN8FOO_ADDR20hc3c6a3fb783d8c08saaE' in file 'foo.0.o'
/usr/bin/ld.gold: removing unused section from '.data' in file '/usr/lib/x86_64-linux-gnu/libc_nonshared.a(elf-init.oS)'
/usr/bin/ld.gold: removing unused section from '.bss' in file '/usr/lib/x86_64-linux-gnu/libc_nonshared.a(elf-init.oS)'
/usr/bin/ld.gold: removing unused section from '.text' in file '/usr/lib/gcc/x86_64-linux-gnu/4.8/crtendS.o'
/usr/bin/ld.gold: removing unused section from '.data' in file '/usr/lib/gcc/x86_64-linux-gnu/4.8/crtendS.o'
/usr/bin/ld.gold: removing unused section from '.bss' in file '/usr/lib/gcc/x86_64-linux-gnu/4.8/crtendS.o'
/usr/bin/ld.gold: removing unused section from '.text' in file '/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crtn.o'
/usr/bin/ld.gold: removing unused section from '.data' in file '/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crtn.o'
/usr/bin/ld.gold: removing unused section from '.bss' in file '/usr/lib/gcc/x86_64-linux-gnu/4.8/../../../x86_64-linux-gnu/crtn.o'

Here we see that our "used symbol" _ZN3foo20hea4f1d23bdee9020vaaE was actually stripped from the final artifact because --gc-sections removes internal symbols that aren't used anywhere. As a result, this attribute is largely just an LLVM-ism rather than a all-the-way-to-the-linker-ism. I believe @mahkoh's examples in this RFC work because the linker must specially recognize the .init_array section and consider all referenced symbols as "used" from there.

This seems like it's at least a downside of this RFC, although it's one that I think it would also be interesting to explore if we could fix as well.

---

The current state of affairs is that we've also stabilized very little with respect to this sort of fine-grained control over linkage and symbol names, which is relatively intentional. That's not to say the set could expand, though! This largely just means that there's not many other pieces of the system we need to consider the interaction with, but that set may grow over time.

I'd personally probably be mostly interested in pursuing this as a nightly feature for now. I'm sure if there are that many use cases for an attribute like this (it looks like .init_array is perhaps one here), but it may be worth getting some more examples before pushing for stabilization in the long run. It may be the case that no practical usage runs into the "bug" I found above (such as the examples in this RFC).

[mahkoh]

@alexcrichton My mistake was that I misunderstood this part of the LLVM docs:

On some targets, the code generator must emit a directive to the assembler or object file to prevent the assembler and linker from molesting the symbol.

I thought this referred to LLVM itself emitting such a directive on certain targets but maybe it is the code that generates the IR that has to do this. I'll investigate what clang does here.

[alexcrichton]

Oh interesting, so according to the docs as well:

then the compiler, assembler, and linker are required to treat the symbol as if there is a reference to the symbol that it cannot see

I wonder if we're perhaps invoking the linker in a weird way? Or maybe it's just not designed to work in tandem with --gc-sections?

It looks like LLVM also has a llvm.compiler.used variable which is specifically meant for just LLVM, not for linkers, but that's likely not what we want here.

[mahkoh]

Both GCC and clang throw __attribute__((used)) symbols away when "-Wl,--gc-sections -ffunction-sections" is passed. The GCC docs say that "used" implies that a symbol is emitted but it doesn't talk about whether the linker can remove the symbol.

If you're using a linker script you can tell the linker to keep a specific section. I haven't found a way to keep a section via a command line option.

[mahkoh]

I don't think there are many uses for symbols are are not referenced anywhere at the linking stage. As long as this this behavior is documented, there shouldn't be an issue.

[arcnmx]

Agreed. The main purpose of the attribute would be to ensure that an internal symbol makes it all the way to the linker. At that point it's free to throw it out if it's truly unused; the linker script is what determines that.

[mahkoh]

It's possible to keep them by putting them all in a section that is used:

#![feature(linkage)]

#[export_name = "llvm.used"]
#[linkage = "appending"]
static mut _FOO: [*const u8; 1] = [
    &foo as *const _ as *const _
];

#[link_section = ".text"]
fn foo() {}

#[link_section = ".text"]
fn main() { }

This is probably good enough for those rare cases that want this behavior.

[alexcrichton]

I'd personally still be a bit wary to give this such a small name like #[used] (as it's not actually what happens), but I'd be more in favor of a name like #[rustc_used] which then used the llvm.compiler.used global as an implementation detail. That seems a bit clearer to me in terms of we can document it as "the compiler will not eliminate this symbol, but other components might", and the name lends itself well to that.

[mahkoh]

(as it's not actually what happens),

What is "it"?

That seems a bit clearer to me in terms of we can document it as "the compiler will not eliminate this symbol, but other components might", and the name lends itself well to that.

The same can be done with a #[used] attribute. What one might do is make #[used] more general (here called #[preserved]):

• #[preserved(none)] does not guarantee that the symbol is preserved or even generated at any point. This is the default.
• #[preserved(compiler)] guarantees that the symbol is not removed by the compiler.
• #[preserved(linker)] guarantees that the symbol is not removed by the linker.
• #[preserved(all)] guarantees that the symbol is not removed by any part of the toolchain.

#[preserved] is an alias for #[preserved(compiler)]. In general #[preserved(X)] means that the symbol is preserved up to at least X in the toolchain.

[arcnmx]

I like preserve (preserved is strange being past-tense), it's longer than used and not quite as ambiguous; it states its purpose a bit better. preserve_symbol could also work if we want to be super explicit.

[alexcrichton]

I'd be a little wary to overdesign a feature like this as it may be difficult to actually guarantee that all the various combinations and such actually work. I'd also want to have a strong motivation for each one before we added it.

I do agree, however, that preserved seems like a better name than used. We could perhaps only add something like preserved(compiler) for now and just leave room for expansion in the future.

[mahkoh]

Updated.

[nikomatsakis]

I too prefer #[preserved] to #[used], though I think I like
#[preserve] a smidgen more (I like bossing my compiler around). Anyhow
"preserve" (or "preserved") feels less generic than "used".

On Sat, Jan 16, 2016 at 6:37 PM, mahkoh [email protected] wrote:

Updated.

—
Reply to this email directly or view it on GitHub
#1459 (comment).

[mahkoh]

The RFC uses the imperative form because it is also used for inline.

[mahkoh]

Updated.

[ArtemGr]

Sorry if it's not entirely on topic, but could the #[used] attribute be extended to local variables, such as lock guards? Lately Rust (or LLVM) has been optimizing away the ProfileGuard variables of the hierarchical profiler (https://github.com/cmr/hprof). I wonder if something like #[used] could be used to prevent the compiler from dropping the variables early, prolonging instead their lifetime to that of their scope.

[mahkoh]

That's an orthogonal issue that can already be solved with fences.

[alexcrichton]

I think that this drawbacks section should probably at least mention that "may be discarded by the linker" can in some cases be confusing. In annotating a symbol as #[preserve] one might expect it to be preserved but it may not make it all the way through.

[mahkoh]

One might expect that #[inline] inlines the function but that's not true either.

[mahkoh]

Updated.

[alexcrichton]

🔔 This RFC is now entering its final comment period 🔔

The tools team discussed this RFC yesterday and the current feeling is that we're likely to not merge this for now. While a number of possibilities have all been discussed here they all suffer from one complication or another unfortunately, and the general story around what symbols end up as what in Rust is relatively vague today as well (so it's tough to provide attributes to control them at an even finer grain).

[brson]

I don't have much opinion on this. It looks reasonable to me.

[nrc]

I don't feel strongly whether we should merge or not, however, istm that most of the sentiment on the thread has been positive and that there is clearly some need for something in this space. So, if we don't merge, I think we should make some effort to spec the details that need to be spec'ed so that our symbol story is not "relatively vague" and we can then progress either merging this RFC or a new one.

[alexcrichton]

Ok, the tools team got a chance to talk about this RFC the other day, and the conclusion was that we're going to close this for now.

Overall our symbol story is pretty vague today and is largely "best effort", so it's difficult to extend it as there's not already a set of documented guarantees. Additionally, once LTO is thrown into the mix, it adds quite a few complications to our symbol visibility story as well. This RFC is also lacking some details with the relationship to LTO and such, so it's not quite fully fleshed out as-is.

Today we approximate symbol visibility on a "best-effort" kind of basis with pub and whether it reaches the root or not. There's also a number of heuristics around #[no_mangle] and extern as well to try to hide as many symbols as possible while still allowing ABI details to get exposed. This ends up all being complicated with LTO as well, but it normally should work.

In other words, though, we should have existing mechanisms to guarantee a symbols makes it to the linker such as:

• pub
• #[no_mangle] and/or extern mixed with pub
• #[linkage], which while unstable, should work

Given all that, to merge an RFC like this we felt that there would want to be a more comprehensive design for our symbols story before adding pieces here and there, so closing.

[japaric]

I've send a PR implementing this in rust-lang/rust#39987. I'd like to hear about what use cases people have in mind for this (apart from the one mentioned in the PR description). Please leave a comment on that PR.