Try to use membarrier on FreeBSD

[SamB]

Currently, you define real wrappers for membarrier on Linux (and wherever you randomly pick up a definition for __NR_membarrier), and everywhere else (except maybe Windows) you define fake wrappers that just fails¹.

Then, if membarrier doesn't claim to support MEMBARRIER_CMD_PRIVATE_EXPEDITED (including the case where we don't know how to call it on a given OS), the calling code to falls back to a hack using mlock(2) and mprotect(2), which unfortunately does not work on ARM/ARM64 Linux or FreeBSD (Apple don't distribute the relevant sources so I have no clue about Mac running on Apple Silicon - not even for x86-64 programs!).

This is a bit hard to spot because it's in two pieces:

1. An explicitly Linux-only bit makes sure you have __NR_membarrier:

   runtime/src/coreclr/pal/src/thread/process.cpp

   Lines 66 to 84 in f196875

   	#ifdef __linux__
   	#include <sys/syscall.h> // __NR_membarrier
   	// Ensure __NR_membarrier is defined for portable builds.
   	# if !defined(__NR_membarrier)
   	# if defined(__amd64__)
   	# define __NR_membarrier 324
   	# elif defined(__i386__)
   	# define __NR_membarrier 375
   	# elif defined(__arm__)
   	# define __NR_membarrier 389
   	# elif defined(__aarch64__)
   	# define __NR_membarrier 283
   	# elif defined(__loongarch64)
   	# define __NR_membarrier 283
   	# else
   	# error Unknown architecture
   	# endif
   	# endif
   	#endif

2. The actual wrappers
   

   runtime/src/coreclr/pal/src/thread/process.cpp

   Lines 146 to 153 in f196875

   	//
   	// Helper membarrier function
   	//
   	#ifdef __NR_membarrier
   	# define membarrier(...) syscall(__NR_membarrier, __VA_ARGS__)
   	#else
   	# define membarrier(...) -ENOSYS
   	#endif

But this misses FreeBSD, which has membarrier(2) too. Apparently they have an actual stub in libc, but have a syscall function too. The stub is declared in sys/membarrier.h and the system call number in sys/syscall.h:

#define	SYS_membarrier	584

The constants are:

enum membarrier_cmd {
	MEMBARRIER_CMD_QUERY =				0x00000000,
	MEMBARRIER_CMD_GLOBAL =				0x00000001,
	MEMBARRIER_CMD_SHARED =				MEMBARRIER_CMD_GLOBAL,
	MEMBARRIER_CMD_GLOBAL_EXPEDITED =		0x00000002,
	MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED =	0x00000004,
	MEMBARRIER_CMD_PRIVATE_EXPEDITED =		0x00000008,
	MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED =	0x00000010,
	MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE =	0x00000020,
	MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE =	0x00000040,

	/*
	 * RSEQ constants are defined for source compatibility but are
	 * not yet supported, MEMBARRIER_CMD_QUERY does not return
	 * them in the mask.
	 */
	MEMBARRIER_CMD_PRIVATE_EXPEDITED_RSEQ =		0x00000080,
	MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_RSEQ = 0x00000100,
};

which appear to match the values for Linux:

runtime/src/coreclr/pal/src/thread/process.cpp

Lines 155 to 165 in f196875

	enum membarrier_cmd
	{
	MEMBARRIER_CMD_QUERY = 0,
	MEMBARRIER_CMD_GLOBAL = (1 << 0),
	MEMBARRIER_CMD_GLOBAL_EXPEDITED = (1 << 1),
	MEMBARRIER_CMD_REGISTER_GLOBAL_EXPEDITED = (1 << 2),
	MEMBARRIER_CMD_PRIVATE_EXPEDITED = (1 << 3),
	MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED = (1 << 4),
	MEMBARRIER_CMD_PRIVATE_EXPEDITED_SYNC_CORE = (1 << 5),
	MEMBARRIER_CMD_REGISTER_PRIVATE_EXPEDITED_SYNC_CORE = (1 << 6)
	};

(Curiously, they don't actually implement Linux compatibility for this syscall despite using the same flag bits.)

Note: for some reason, the same exact code appears in https://github.com/dotnet/runtime/blob/f196875c5d0441364f6538d2dd8fc4e3eb1ff108/src/coreclr/gc/unix/gcenv.unix.cpp, though with different line numbers.

Footnotes

1. Which is a great way to avoid boring, repetitive, and therefore bug prone code, though I note that glibc's syscall() function would set errno = ENOSYS and return -1 rather than returning -ENOSYS if the syscall is not supported. This would only truly matter if you actually had callers that distinguished between negative values or read errno, which you don'tx ↩

[dotnet-policy-service]

Tagging subscribers to this area: @mangod9
See info in area-owners.md if you want to be subscribed.

[janvorli]

@SamB thank you for pointing this out. Just FYI, we have actually removed the __NR_membarrier definitions in a PR last week. These were for backwards compatibility with the oldest supported Linux distro. And we no longer support a distro where these would not be defined.

The FreeBSD port is a community effort, so let me cc some of the folks that work on it:
@Thefrank, @sec - do you know if the membarrier support on FreeBSD is sufficient to achieve what .NET needs it for?

[sec]

That's out of my knowledge - before when I tried Linux emu layer on FreeBSD, it was failing becuase of missing that syscall, but according to https://www.freebsd.org/status/report-2021-10-2021-12/membarrier-rseq/ there's no plan for 100% match

[Thefrank]

as @sec pointed out: membarrier in FreeBSD is a clean-room implementation and not an exact 1:1 reproduction as it serves to help those using the linuxcompat service.

[jkotas]

Looking at the FreeBSD implementation of membarrier, it should be sufficient for what .NET runtime needs. Fixing this issue should about cleaning up the conditions around membarrier calls so that they get called on FreeBSD.