Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BPF] Add load-acquire and store-release instructions under -mcpu=v4 #108636

Open
wants to merge 3 commits into
base: main
Choose a base branch
from
Open

[BPF] Add load-acquire and store-release instructions under -mcpu=v4 #108636

wants to merge 3 commits into from

Commits on Oct 17, 2024

  1. [BPF] Rename isST*() and isLD*() functions in BPFMISimplifyPatchable.… 

    …cpp (NFC)

We are planning to add load (specifically, atomic acquiring load, or
"load-acquire") instructions under the STX instruction class.  To make
that easier, rename the isST*() and isLD*() helper functions based on
what the instructions actually do, rather than their instruction class.
    @peilin-ye
    peilin-ye committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    32cc1b2 View commit details
    Browse the repository at this point in the history

  2. [BPF] Add load-acquire and store-release instructions under -mcpu=v4 

    As discussed in [1], introduce BPF instructions with load-acquire and
store-release semantics under -mcpu=v4.

A "load-acquire" is a BPF_STX | BPF_ATOMIC instruction with the 'imm'
field set to BPF_LOAD_ACQ (0x1, acquiring atomic load).  Similarly, a
"store-release" is a BPF_STX | BPF_ATOMIC instruction with the 'imm'
field set to BPF_STORE_REL (0xb, releasing atomic store).

Unlike existing atomic operations that only support BPF_W (32-bit) and
BPF_DW (64-bit) size modifiers, load-acquires and store-releases also
support BPF_B (8-bit) and BPF_H (16-bit).  An 8- or 16-bit load-acquire
zero-extends the value before writing it to a 32-bit register, just like
ARM64 instruction LDAPRH and friends.

As an example, for -march=bpfel:

  long foo(long *ptr) {
      return __atomic_load_n(ptr, __ATOMIC_ACQUIRE);
  }

foo() can be compiled to:

  db 10 00 00 10 00 00 00  r0 = load_acquire((u64 *)(r1 + 0x0))
  95 00 00 00 00 00 00 00  exit

  opcode (0xdb): BPF_ATOMIC | BPF_DW | BPF_STX
  imm (0x00000010): BPF_LOAD_ACQ

Similarly:

  void bar(short *ptr, short val) {
      __atomic_store_n(ptr, val, __ATOMIC_RELEASE);
  }

bar() can be compiled to:

  cb 21 00 00 b0 00 00 00  store_release((u16 *)(r1 + 0x0), w2)
  95 00 00 00 00 00 00 00  exit

  opcode (0xcb): BPF_ATOMIC | BPF_H | BPF_STX
  imm (0x000000b0): BPF_STORE_REL

Inline assembly is also supported.  For example:

  asm volatile("%0 = load_acquire((u64 *)(%1 + 0x0))" :
               "=r"(ret) : "r"(ptr) : "memory");

Add two macros, __BPF_FEATURE_LOAD_ACQUIRE and
__BPF_FEATURE_STORE_RELEASE, to let developers detect these new features
in source code.  They can also be disabled using two new llc options,
-disable-load-acquire and -disable-store-release, respectively.

Also use ACQUIRE or RELEASE if user requested weaker memory orders
(RELAXED or CONSUME) until we actually support them.  Requesting a
stronger memory order (i.e. SEQ_CST) will cause an error.

[1] https://lore.kernel.org/all/[email protected]/
    @peilin-ye
    peilin-ye committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    ee7836e View commit details
    Browse the repository at this point in the history

  3. [BPF] Improve error message for seq_cst atomic load and store 

    Sequentially consistent (seq_cst) atomic load and store are not
supported yet for BPF.  Right now, calling __atomic_{load,store}{,_n}()
with __ATOMIC_SEQ_CST will cause an error:

  $ cat bar.c
  int foo(int *ptr) { return __atomic_load_n(ptr, __ATOMIC_SEQ_CST); }
  $ clang --target=bpf -mcpu=v4 -c bar.c > /dev/null
  fatal error: error in backend: Cannot select: t8: i32,ch = AtomicLoad<(load seq_cst (s32) from %ir.0)> t7:1, t7
  ...

Which isn't very useful.  Just like commit 379d908 ("BPF: provide
better error message for unsupported atomic operations"), make it
generate an error message saying that the requested operation isn't
supported, before triggering that "fatal error":

  $ clang --target=bpf -mcpu=v4 -c bar.c > /dev/null
  bar.c:1:5: error: sequentially consistent (seq_cst) atomic load is not supported
    1 | int foo(int *ptr) { return __atomic_load_n(ptr, __ATOMIC_SEQ_CST); }
      |     ^
  ...
    @peilin-ye
    peilin-ye committed Oct 17, 2024
    Configuration menu
    Copy the full SHA
    827f517 View commit details
    Browse the repository at this point in the history