Consider open-coding subword atomics or move to libgcc

[sorear]

The problem

Code which uses __atomic_xxx builtins or C11 atomics requires to be linked with -latomic on RISC-V, but not on other common architectures (including but not limited to ARM, x86).

Example code:

unsigned char ach;
int main() {
  __atomic_fetch_add(&ach,1,__ATOMIC_RELAXED);
  return 0;
}

Testing with the tool at gcc.godbolt.org:

• x86, x86_64, ARM, and ARM64 have native support for byte atomics
• PowerPC, MIPS, MIPS64 open-codes a full word LL/increment one byte/SC loop
• MSP430 generates a call to an external helper function
• the AVR compiler is too old to have __atomic_fetch_add

Option 1: Use existing libgcc helper

Modify gcc to generate a call to __sync_fetch_and_add_1 instead of __atomic_fetch_add_1, and similarly for other size-1 and size-2 atomics.

Advantages: code size, avoids polluting the libgcc ABI with weirdness

Option 2: Move helpers from libatomic to libgcc

Make libgcc provide a subset of the libatomic names.

Advantages: Probably the easiest in the short term

Disadvantages: Creates an ABI mess that we'll probably regret later, especially for things like compiler-rt (LLVM's compatible reimplementation of libgcc)

Option 3: Inline code generation

As MIPS/MIPS64/PowerPC are already doing.

Advantages: sidesteps ABI issues entirely

Option 4: Add --as-needed -latomic to the spec file

Arguably gcc should be doing this on all platforms, _Atomic(char[32]) is in C11 and requiring a non-POSIX -l flag to use standard C features is quite dodgy.

Disadvantages: Depletes the RISC-V weirdness budget

Option 5: Do nothing and change the build scripts of affected software

So far this includes QEMU and the Boehm GC. I haven't analyzed all of the Fedora build failures so it might have caused others.

Disadvantages: Depletes the RISC-V weirdness budget much more since it affects non-compiler software

[kito-cheng]

Option 3 + 1

[aswaterman]

• Option 1 seems pretty reasonable. It's theoretically less performant, since the __sync variants don't support the relaxed memory models, but in practice it's probably irrelevant for these LR/SC sequences.
• I dislike Option 2 for the reason you mentioned.
• Option 3 sounds more like an optimization you'd enable only for -O3, as it will bloat the code--especially for ADD/SWAP/MIN/MAX, which need to be turned into lengthy LR/SC sequences. For AND/OR/XOR, it's not so bad.
• Option 4 seems pretty reasonable. For few users does it deplete the weirdness budget.
• I dislike Option 5 for the reason you mentioned.

[aswaterman]

I lean towards Option 4. I don't see why libatomic exists at all if it's considered weird to link against it.

[sorear]

@aswaterman I'm only calling option 4 "weird" because changing the default linker command line is something that GCC should do across the board, not one architecture at a time.

(libatomic exists (AIUI) primarily to support double-word and wider atomics using the ifunc mechanism. When the T extension is finalized (or a DW-CAS extension, potentially), libatomic can be modified to use it for __atomic_compare_exchange_16. That way you can write an algorithm which uses DW-CAS, have it run on DW-CAS capable hardware, and also run (with lower concurrency due to contention the lock table) on hardware without DW-CAS.

Dynamically linking libatomic is required for ifuncs to work, and also allows entirely new instructions to be used without recompiling programs. libgcc, by contrast, is statically linked by default.

For 1, 2 byte atomics the fallback code is already lock free so being able to replace it at runtime is much less valuable. If you had a chip with byte/word AMOs, the savings from using them would probably be of the same order as the PLT call overhead.)

[aswaterman]

@sorear understood. I don't have much of a problem being an avant garde linker-invoker, but it does seem this should eventually change globally. Until then, would you mind submitting a PR implementing option 4?

[sorear]

@aswaterman If you don't think 4 will cause upstreaming problems ("Why did you change that? The link order is supposed to be the same everywhere"), then I'm fine with it. Can do a PR shortly. (Option 3 might still make sense at -O3, at least because it clobbers fewer registers than a shared library function call; for another time.)

[aswaterman]

@sorear hard to know. Maybe it will nudge the maintainers towards an appropriate global fix. Regardless, I'm inclined to cross that bridge when we get there. If the maintainers object, we can fall back to option 1. (I concur w.r.t. Option 3.)

[sorear]

#15 is another bit of related weirdness which does not seem to materially affect software. I think options 1 and 3 would resolve #15, the others wouldn't.

[aswaterman]

2c4857d implements option 4.

[rishikhan]

where is this libatomic? I don't see it in /opt/riscv on my system after installing riscv-gnu-toolchain.

[jim-wilson]

libatomic requires some OS support. It is built for linux targets. It is not built for bare metal embedded targets.

[rishikhan]

I’m curious as to what the OS support is? It would seem like a simple spinlock would give you the semantics necessary completely in user space.

…

On Dec 18, 2017, at 11:20 AM, Jim Wilson ***@***.***> wrote: libatomic requires some OS support. It is built for linux targets. It is not built for bare metal embedded targets. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#12 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHJS6RNwA1Ija5SH3zt6jLidbXuJs828ks5tBpDYgaJpZM4Kgy9p>.

[jim-wilson]

The posix support uses pthread_mutex_lock and pthread_mutex_unlock. The mingw (Windows) support uses CreateMutex, WaitForSingleObject, and ReleaseMutex. It would be possible to handle locks other ways. No one has written code to do that.

I would expect that the simplest atomic operations get expanded inline, and you only call libatomic for operations that can't easily be open coded.

[rishikhan]

Yes. I've seen them inline...

…

On Dec 18, 2017 2:10 PM, "Jim Wilson" ***@***.***> wrote: The posix support uses pthread_mutex_lock and pthread_mutex_unlock. The mingw (Windows) support uses CreateMutex, WaitForSingleObject, and ReleaseMutex. It would be possible to handle locks other ways. No one has written code to do that. I would expect that the simplest atomic operations get expanded inline, and you only call libatomic for operations that can't easily be open coded. — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#12 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHJS6Q_TBe3icmLqYYl1WPpafgK-ycrBks5tBri0gaJpZM4Kgy9p> .

[sorear]

This appears to have regressed: https://groups.google.com/a/groups.riscv.org/d/msgid/sw-dev/5fc5d933-7312-465e-85f0-1aa3a00e17d1%40groups.riscv.org

[rishikhan]

It looks like that person is running -lpthread instead of -pthread. Thus, -latomic is not linked.

…

On Feb 10, 2018 3:51 AM, "sorear" ***@***.***> wrote: This appears to have regressed: https://groups.google.com/a/ groups.riscv.org/d/msgid/sw-dev/5fc5d933-7312-465e-85f0- 1aa3a00e17d1%40groups.riscv.org — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#12 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHJS6XMx6D8BDoQiCxBS2IuWazuz8F3Rks5tTVh3gaJpZM4Kgy9p> .

[jyknight]

The intent of libatomic has ben that for a given object size/alignment, the compiler should either call libatomic's _atomic* for all the operations, or call it for no operations, rather than mix-and-match per operation.

Concretely, the compiler should either call __atomic_* functions for all the 1-byte atomic operations (load, fetch-add, store, etc), or it should call them for none. Since in RISCV with A extension allows these to be implemented lock-free, it should not be calling __atomic_* for any of them.

The valid choices are to call an external helper function (which can be implemented in the static part of libgcc, since it has no state), or open-code it.

I'd suggest open-coding, but option 1 would be fine too.

[jim-wilson]

The plan is to open code all of the atomics, but there is no volunteer to do the work, and unclear when it will happen.

[alexfanqi]

Since glibc 2.34, pthread is builtin and recent cmake FindThreads module find -pthread is unneccessary therefore doesn't include -pthread flag in linking. I have to condition on CMAKE_USE_PTHREADS_INIT to explicitly pass -pthread as a workaround. In this case, does 2c4857d or https://github.com/gcc-mirror/gcc/blob/041cfa0ce44d4c207903d41e6eabccdab2dfa90b/gcc/config/riscv/linux.h#L42 still valid?

[jim-wilson]

Development work happens upstream. FSF GCC developers do not follow this repo. They follow the upstream repo. This site is really only useful for helping newbies. Anything development related needs to be discussed upstream. Long term, something needs to happen, but it won't happen here, it will happen upstream.

Yes, the riscv/linux.h line 42 is still valid. GCC supports more than one version of glibc, and gcc docs still say that -pthread is required. Also note that not all UNIX systems use glibc, so we can't change how things work just because of a glibc change, unless perhaps that change is conditional on use of glibc and the glibc version.

It is possible for distros to work around the problem. I think debian puts a AS_NEEDED (libatomic.so) in libc.so for instance. Or a distro could force libatomic to be included in libpthread. There may also be other ways to work around the problem.

[eli-schwartz]

The Meson build system now has a feature request that we add a new feature just to enable passing -latomic into packages in accordance with "Option 5: Do nothing and change the build scripts of affected software".

This feels like a pretty high weirdness budget even compared to the general case of "non-compiler software". I wonder if a solution can be found...

2c4857d implements option 4.

This was apparently reverted to only be invoked during pthread as part of a giant squashed upstream merge beyond which nothing is clearly visible? Anyway, it seems this ticket is no longer actually completed.