Wrong behavior on s390x #86
Comments
|
Hi! That's probably because you're the first s390x user :) Note the following: vs. The operands are switched: the library thinks it's emitting Can you give me some more information about your system (kernel, userspace)? Is it running in big endian mode? |
|
hi @lmb $ cat /etc/os-release
NAME=Fedora
VERSION="31 (Thirty One)"
ID=fedora
VERSION_ID=31
VERSION_CODENAME=""
PLATFORM_ID="platform:f31"
PRETTY_NAME="Fedora 31 (Thirty One)"
ANSI_COLOR="0;34"
LOGO=fedora-logo-icon
CPE_NAME="cpe:/o:fedoraproject:fedora:31"
HOME_URL="https://fedoraproject.org/"
DOCUMENTATION_URL="https://docs.fedoraproject.org/en-US/fedora/f31/system-administrators-guide/"
SUPPORT_URL="https://fedoraproject.org/wiki/Communicating_and_getting_help"
BUG_REPORT_URL="https://bugzilla.redhat.com/"
REDHAT_BUGZILLA_PRODUCT="Fedora"
REDHAT_BUGZILLA_PRODUCT_VERSION=31
REDHAT_SUPPORT_PRODUCT="Fedora"
REDHAT_SUPPORT_PRODUCT_VERSION=31
PRIVACY_POLICY_URL="https://fedoraproject.org/wiki/Legal:PrivacyPolicy"
$ uname -rm
5.3.7-301.fc31.s390x s390xyes, it is running in bigendian mode. Probably, there's some endianness issue somewhere in the code. Having a look |
|
Yeah, I think https://github.com/cilium/ebpf/blob/master/asm/instruction.go#L435-L445 needs to be swapped for big endian. Annoying, since the registers are encoded as an uint8, which binary.ByteOrder doesn't support natively. |
|
@lmb I tried a quick hack git diff
diff --git a/asm/instruction.go b/asm/instruction.go
index 644de42..ae6f3a0 100644
--- a/asm/instruction.go
+++ b/asm/instruction.go
@@ -77,7 +77,7 @@ func (ins Instruction) Marshal(w io.Writer, bo binary.ByteOrder) (uint64, error)
bpfi := bpfInstruction{
ins.OpCode,
- newBPFRegisters(ins.Dst, ins.Src),
+ newBPFRegisters(ins.Src, ins.Dst),
ins.Offset,
cons,
}
$ go test -run TestProgramRun
PASS
ok github.com/cilium/ebpf 0.002sand it worked. You were right! |
|
Do all of the other tests pass on s390x? |
|
@lmb unfortunately not. go test
--- FAIL: TestLoadCollectionSpec (0.00s)
--- FAIL: TestLoadCollectionSpec/loader-clang-6.0.elf (0.00s)
elf_reader_test.go:85: xdp_prog:
0: MovImm dst: r1 imm: 0
1: StXMemW dst: rfp src: r1 off: -4 imm: 0
2: MovImm dst: r1 imm: 1
3: StXMemW dst: rfp src: r1 off: -8 imm: 0
4: MovImm dst: r1 imm: 2
5: StXMemW dst: rfp src: r1 off: -12 imm: 0
6: MovReg dst: r2 src: rfp
7: AddImm dst: r2 imm: -4
8: LoadMapPtr dst: r1 fd: -1 <hash_map>
10: Call FnMapLookupElem
11: MovReg dst: r2 src: rfp
12: AddImm dst: r2 imm: -8
13: LoadMapPtr dst: r1 fd: -1 <hash_map2>
15: Call FnMapLookupElem
16: MovReg dst: r2 src: rfp
17: AddImm dst: r2 imm: -12
18: LoadMapPtr dst: r1 fd: -1 <hash_map2>
20: Call FnMapLookupElem
21: MovImm dst: r1 imm: 1
22: Call -1 <global_fn>
23: AddImm dst: r0 imm: 1
24: Exit
global_fn2:
25: MovReg dst: r0 src: r1
26: Exit
global_fn:
27: MovReg dst: r6 src: r1
28: Call 10
29: MovReg dst: r7 src: r0
30: MovReg dst: r1 src: r6
31: Call -7
32: MovReg dst: r8 src: r0
33: AddReg dst: r8 src: r7
34: MovReg dst: r1 src: r6
35: Call -1 <global_fn3>
36: AddReg dst: r8 src: r0
37: MovReg dst: r0 src: r8
38: Exit
static_fn:
39: MovReg dst: r0 src: r1
40: Exit
global_fn3:
41: AddImm dst: r1 imm: 1
42: MovReg dst: r0 src: r1
43: Exit
elf_reader_test.go:94: program xdp_prog: can't load program: errno 524: 0: (b7) r1 = 0
1: (63) *(u32 *)(r10 -4) = r1
2: (b7) r1 = 1
3: (63) *(u32 *)(r10 -8) = r1
4: (b7) r1 = 2
5: (63) *(u32 *)(r10 -12) = r1
6: (bf) r2 = r10
7: (07) r2 += -4
8: (18) r1 = 0x1fbd42800
10: (85) call bpf_map_lookup_elem#1
11: (bf) r2 = r10
12: (07) r2 += -8
13: (18) r1 = 0x1fbd42000
15: (85) call bpf_map_lookup_elem#1
16: (bf) r2 = r10
17: (07) r2 += -12
18: (18) r1 = 0x1fbd42000
20: (85) call bpf_map_lookup_elem#1
21: (b7) r1 = 1
22: (85) call pc+4
caller:
R10=fp0 fp-8=mmmmmmmm fp-16=mmmm????
callee:
frame1: R1_w=invP1 R10=fp0
27: (bf) r6 = r1
28: (85) call pc+10
caller:
frame1: R6_w=invP1 R10=fp0
callee:
frame2: R1_w=invP1 R10=fp0
39: (bf) r0 = r1
40: (95) exit
returning from callee:
frame2: R0_w=invP1 R1=invP1 R10=fp0
to caller at 29:
frame1: R0_w=invP1 R6=invP1 R10=fp0
from 40 to 29: frame1: R0_w=invP1 R6=invP1 R10=fp0
29: (bf) r7 = r0
30: (bf) r1 = r6
31: (85) call pc-7
caller:
frame1: R6=invP1 R7_w=invP1 R10=fp0
callee:
frame2: R1_w=invP1 R10=fp0
25: (bf) r0 = r1
26: (95) exit
returning from callee:
frame2: R0_w=invP1 R1_w=invP1 R10=fp0
to caller at 32:
frame1: R0_w=invP1 R6=invP1 R7_w=invP1 R10=fp0
from 26 to 32: frame1: R0_w=invP1 R6=invP1 R7_w=invP1 R10=fp0
32: (bf) r8 = r0
33: (0f) r8 += r7
34: (bf) r1 = r6
35: (85) call pc+5
caller:
frame1: R6=invP1 R7=invP1 R8=invP2 R10=fp0
callee:
frame2: R1=invP1 R10=fp0
41: (07) r1 += 1
42: (bf) r0 = r1
43: (95) exit
returning from callee:
frame2: R0_w=invP2 R1_w=invP2 R10=fp0
to caller at 36:
frame1: R0_w=invP2 R6=invP1 R7=invP1 R8=invP2 R10=fp0
from 43 to 36: frame1: R0_w=invP2 R6=invP1 R7=invP1 R8=invP2 R10=fp0
36: (0f) r8 += r0
37: (bf) r0 = r8
38: (95) exit
returning from callee:
frame1: R0_w=invP4 R6=invP1 R7=invP1 R8_w=invP4 R10=fp0
to caller at 23:
R0_w=invP4 R10=fp0 fp-8=mmmmmmmm fp-16=mmmm????
from 38 to 23: R0_w=invP4 R10=fp0 fp-8=mmmmmmmm fp-16=mmmm????
23: (07) r0 += 1
24: (95) exit
JIT doesn't support bpf-to-bpf calls
processed 41 insns (limit 1000000) max_states_per_insn 0 total_states 3 peak_states 3 mark_read 1
--- FAIL: TestLoadCollectionSpec/loader-clang-7.elf (0.00s)
elf_reader_test.go:85: xdp_prog:
0: MovImm dst: r1 imm: 0
1: StXMemW dst: rfp src: r1 off: -4 imm: 0
2: MovImm dst: r1 imm: 1
3: StXMemW dst: rfp src: r1 off: -8 imm: 0
4: MovImm dst: r1 imm: 2
5: StXMemW dst: rfp src: r1 off: -12 imm: 0
6: MovReg dst: r2 src: rfp
7: AddImm dst: r2 imm: -4
8: LoadMapPtr dst: r1 fd: -1 <hash_map>
10: Call FnMapLookupElem
11: MovReg dst: r2 src: rfp
12: AddImm dst: r2 imm: -8
13: LoadMapPtr dst: r1 fd: -1 <hash_map2>
15: Call FnMapLookupElem
16: MovReg dst: r2 src: rfp
17: AddImm dst: r2 imm: -12
18: LoadMapPtr dst: r1 fd: -1 <hash_map2>
20: Call FnMapLookupElem
21: MovImm dst: r1 imm: 1
22: Call -1 <global_fn>
23: AddImm dst: r0 imm: 1
24: Exit
global_fn2:
25: MovReg dst: r0 src: r1
26: Exit
global_fn:
27: MovReg dst: r6 src: r1
28: Call 10
29: MovReg dst: r7 src: r0
30: MovReg dst: r1 src: r6
31: Call -7
32: MovReg dst: r8 src: r0
33: AddReg dst: r8 src: r7
34: MovReg dst: r1 src: r6
35: Call -1 <global_fn3>
36: AddReg dst: r8 src: r0
37: MovReg dst: r0 src: r8
38: Exit
static_fn:
39: MovReg dst: r0 src: r1
40: Exit
global_fn3:
41: MovReg dst: r0 src: r1
42: AddImm dst: r0 imm: 1
43: Exit
elf_reader_test.go:94: program xdp_prog: can't load program: errno 524: 0: (b7) r1 = 0
1: (63) *(u32 *)(r10 -4) = r1
2: (b7) r1 = 1
3: (63) *(u32 *)(r10 -8) = r1
4: (b7) r1 = 2
5: (63) *(u32 *)(r10 -12) = r1
6: (bf) r2 = r10
7: (07) r2 += -4
8: (18) r1 = 0x1f84fe800
10: (85) call bpf_map_lookup_elem#1
11: (bf) r2 = r10
12: (07) r2 += -8
13: (18) r1 = 0x1f84f9000
15: (85) call bpf_map_lookup_elem#1
16: (bf) r2 = r10
17: (07) r2 += -12
18: (18) r1 = 0x1f84f9000
20: (85) call bpf_map_lookup_elem#1
21: (b7) r1 = 1
22: (85) call pc+4
caller:
R10=fp0 fp-8=mmmmmmmm fp-16=mmmm????
callee:
frame1: R1_w=invP1 R10=fp0
27: (bf) r6 = r1
28: (85) call pc+10
caller:
frame1: R6_w=invP1 R10=fp0
callee:
frame2: R1_w=invP1 R10=fp0
39: (bf) r0 = r1
40: (95) exit
returning from callee:
frame2: R0_w=invP1 R1=invP1 R10=fp0
to caller at 29:
frame1: R0_w=invP1 R6=invP1 R10=fp0
from 40 to 29: frame1: R0_w=invP1 R6=invP1 R10=fp0
29: (bf) r7 = r0
30: (bf) r1 = r6
31: (85) call pc-7
caller:
frame1: R6=invP1 R7_w=invP1 R10=fp0
callee:
frame2: R1_w=invP1 R10=fp0
25: (bf) r0 = r1
26: (95) exit
returning from callee:
frame2: R0_w=invP1 R1_w=invP1 R10=fp0
to caller at 32:
frame1: R0_w=invP1 R6=invP1 R7_w=invP1 R10=fp0
from 26 to 32: frame1: R0_w=invP1 R6=invP1 R7_w=invP1 R10=fp0
32: (bf) r8 = r0
33: (0f) r8 += r7
34: (bf) r1 = r6
35: (85) call pc+5
caller:
frame1: R6=invP1 R7=invP1 R8=invP2 R10=fp0
callee:
frame2: R1=invP1 R10=fp0
41: (bf) r0 = r1
42: (07) r0 += 1
43: (95) exit
returning from callee:
frame2: R0_w=invP2 R1=invP1 R10=fp0
to caller at 36:
frame1: R0_w=invP2 R6=invP1 R7=invP1 R8=invP2 R10=fp0
from 43 to 36: frame1: R0_w=invP2 R6=invP1 R7=invP1 R8=invP2 R10=fp0
36: (0f) r8 += r0
37: (bf) r0 = r8
38: (95) exit
returning from callee:
frame1: R0_w=invP4 R6=invP1 R7=invP1 R8_w=invP4 R10=fp0
to caller at 23:
R0_w=invP4 R10=fp0 fp-8=mmmmmmmm fp-16=mmmm????
from 38 to 23: R0_w=invP4 R10=fp0 fp-8=mmmmmmmm fp-16=mmmm????
23: (07) r0 += 1
24: (95) exit
JIT doesn't support bpf-to-bpf calls
processed 41 insns (limit 1000000) max_states_per_insn 0 total_states 3 peak_states 3 mark_read 1
--- FAIL: TestLoadCollectionSpec/loader-clang-8.elf (0.00s)
elf_reader_test.go:85: xdp_prog:
0: MovImm dst: r1 imm: 0
1: StXMemW dst: rfp src: r1 off: -4 imm: 0
2: MovImm dst: r1 imm: 1
3: StXMemW dst: rfp src: r1 off: -8 imm: 0
4: MovImm dst: r1 imm: 2
5: StXMemW dst: rfp src: r1 off: -12 imm: 0
6: MovReg dst: r2 src: rfp
7: AddImm dst: r2 imm: -4
8: LoadMapPtr dst: r1 fd: -1 <hash_map>
10: Call FnMapLookupElem
11: MovReg dst: r2 src: rfp
12: AddImm dst: r2 imm: -8
13: LoadMapPtr dst: r1 fd: -1 <hash_map2>
15: Call FnMapLookupElem
16: MovReg dst: r2 src: rfp
17: AddImm dst: r2 imm: -12
18: LoadMapPtr dst: r1 fd: -1 <hash_map2>
20: Call FnMapLookupElem
21: MovImm dst: r1 imm: 1
22: Call -1 <global_fn>
23: AddImm dst: r0 imm: 1
24: Exit
global_fn2:
25: MovReg dst: r0 src: r1
26: Exit
global_fn:
27: MovReg dst: r6 src: r1
28: Call 10
29: MovReg dst: r7 src: r0
30: MovReg dst: r1 src: r6
31: Call -7
32: MovReg dst: r8 src: r0
33: AddReg dst: r8 src: r7
34: MovReg dst: r1 src: r6
35: Call -1 <global_fn3>
36: AddReg dst: r8 src: r0
37: MovReg dst: r0 src: r8
38: Exit
static_fn:
39: MovReg dst: r0 src: r1
40: Exit
global_fn3:
41: MovReg dst: r0 src: r1
42: AddImm dst: r0 imm: 1
43: Exit
elf_reader_test.go:94: invalid argument: magic: 0xeb9f
version: 1
flags: 0x0
hdr_len: 24
type_off: 0
type_len: 424
str_off: 424
str_len: 887
btf_total_size: 1335
[1] TYPEDEF uint32_t type_id=2
[2] INT unsigned int Invalid int_data:20000000
--- FAIL: TestLoadCollectionSpec/loader-clang-9.elf (0.00s)
elf_reader_test.go:85: xdp_prog:
0: LoadMapPtr dst: r1 fd: -1 <hash_map>
2: LoadMapValue dst: r2, fd: -1 off: 0 <.bss>
4: Call FnMapLookupElem
5: LoadMapPtr dst: r1 fd: -1 <hash_map2>
7: LoadMapValue dst: r2, fd: -1 off: 0 <.data>
9: Call FnMapLookupElem
10: LoadMapPtr dst: r1 fd: -1 <hash_map2>
12: LoadMapValue dst: r2, fd: -1 off: 0 <.rodata>
14: Call FnMapLookupElem
15: LoadMapValue dst: r7, fd: -1 off: 4 <.rodata>
17: LdXMemW dst: r1 src: r7 off: 0 imm: 0
18: Call -1 <static_fn>
19: MovReg dst: r6 src: r0
20: LdXMemW dst: r1 src: r7 off: 0 imm: 0
21: Call -1 <global_fn>
22: AddReg dst: r0 src: r6
23: Exit
global_fn2:
24: MovReg dst: r0 src: r1
25: Exit
global_fn:
26: MovReg dst: r6 src: r1
27: Call 10
28: MovReg dst: r7 src: r0
29: MovReg dst: r1 src: r6
30: Call -7
31: MovReg dst: r8 src: r0
32: AddReg dst: r8 src: r7
33: MovReg dst: r1 src: r6
34: Call -1 <global_fn3>
35: AddReg dst: r8 src: r0
36: MovReg dst: r0 src: r8
37: Exit
static_fn:
38: MovReg dst: r0 src: r1
39: Exit
global_fn3:
40: MovReg dst: r0 src: r1
41: AddImm dst: r0 imm: 1
42: Exit
elf_reader_test.go:94: invalid argument: magic: 0xeb9f
version: 1
flags: 0x0
hdr_len: 24
type_off: 0
type_len: 868
str_off: 868
str_len: 916
btf_total_size: 1808
[1] STRUCT (anon) size=40 vlen=5
type type_id=2 bits_offset=0
key type_id=6 bits_offset=64
value type_id=6 bits_offset=128
max_entries type_id=2 bits_offset=192
map_flags type_id=2 bits_offset=256
[2] PTR (anon) type_id=4
[3] INT int Invalid int_data:20000001
--- FAIL: TestLink (0.00s)
linker_test.go:43: 0: MovReg dst: r0 src: r1
1: Call -1 <my_func>
2: Exit
my_func:
3: Call -1 <my_other_func>
4: Exit
my_other_func:
5: LdImmDW dst: r0 imm: 1337
7: Exit
linker_test.go:49: can't load program: errno 524: 0: (bf) r0 = r1
1: (85) call pc+1
caller:
R10=fp0
callee:
frame1: R1=ctx(id=0,off=0,imm=0) R10=fp0
3: (85) call pc+1
caller:
frame1: R10=fp0
callee:
frame2: R1=ctx(id=0,off=0,imm=0) R10=fp0
5: (18) r0 = 0x539
7: (95) exit
returning from callee:
frame2: R0_w=invP1337 R1=ctx(id=0,off=0,imm=0) R10=fp0
to caller at 4:
frame1: R0_w=invP1337 R10=fp0
from 7 to 4: frame1: R0_w=invP1337 R10=fp0
4: (95) exit
returning from callee:
frame1: R0_w=invP1337 R10=fp0
to caller at 2:
R0_w=invP1337 R10=fp0
from 4 to 2: R0_w=invP1337 R10=fp0
2: (95) exit
JIT doesn't support bpf-to-bpf calls
processed 7 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
FAIL
exit status 1
FAIL github.com/cilium/ebpf 0.161s
|
|
But I'm not sure if those tests should be skipped as we get |
|
I tried to recompile the $ clang --version
clang version 9.0.1 (Fedora 9.0.1-2.fc31)
Target: s390x-ibm-linux
Thread model: posix
InstalledDir: /usr/bin
$ cd testdata/
$ make clean
$ make loader-clang-9.elf rewrite.elf invalid_map.elf
$ ls
common.h invalid_map.c invalid_map.elf loader.c loader-clang-9.elf Makefile rewrite.c rewrite.elfNow, I get: go test
--- FAIL: TestLoadCollectionSpec (0.00s)
--- FAIL: TestLoadCollectionSpec/loader-clang-9.elf (0.00s)
elf_reader_test.go:23: Can't parse ELF: file testdata/loader-clang-9.elf: load programs: program global_fn2: can't unmarshal instructions: offset 80: can't relocate instruction: call: global_fn3: incorrect source register
--- FAIL: TestLink (0.00s)
linker_test.go:43: 0: MovReg dst: r0 src: r1
1: Call -1 <my_func>
2: Exit
my_func:
3: Call -1 <my_other_func>
4: Exit
my_other_func:
5: LdImmDW dst: r0 imm: 1337
7: Exit
linker_test.go:49: can't load program: errno 524: 0: (bf) r0 = r1
1: (85) call pc+1
caller:
R10=fp0
callee:
frame1: R1=ctx(id=0,off=0,imm=0) R10=fp0
3: (85) call pc+1
caller:
frame1: R10=fp0
callee:
frame2: R1=ctx(id=0,off=0,imm=0) R10=fp0
5: (18) r0 = 0x539
7: (95) exit
returning from callee:
frame2: R0_w=invP1337 R1=ctx(id=0,off=0,imm=0) R10=fp0
to caller at 4:
frame1: R0_w=invP1337 R10=fp0
from 7 to 4: frame1: R0_w=invP1337 R10=fp0
4: (95) exit
returning from callee:
frame1: R0_w=invP1337 R10=fp0
to caller at 2:
R0_w=invP1337 R10=fp0
from 4 to 2: R0_w=invP1337 R10=fp0
2: (95) exit
JIT doesn't support bpf-to-bpf calls
processed 7 insns (limit 1000000) max_states_per_insn 0 total_states 0 peak_states 0 mark_read 0
FAIL
exit status 1
FAIL github.com/cilium/ebpf 0.175s
|
|
Can you upload the compiled testdata somewhere? No idea how to get cross compiling to work for this. Issues I can see:
We can't do much about the last point, except detect the issue and skip the test. Bummer, I think bpf-2-bpf calls are pretty important. Is there a way to get access to a s390x VM with a recent kernel? |
|
@lmb I might request a VM for the community, but it takes some time. Plus I have to check which distro they support. I may try to get the latest ubuntu distro and upgrade the kernel to the most recent one. Second solution is to boot an emulated s390x Fedora 31 image using qemu. Could you please ping me in the |
|
If you want to work on this, I have the following questions / suggestions:
For testing: as long as there is no big endian CI we might get away with adding some tests that compare some serialized eBPF from the |
|
@lmb about enabling the CI for s390x. There are a couple of issues to do that. The main problem how to get a kernel image for s390x. You're now fetching the kernel from https://github.com/cilium/ci-kernels. It is of course possible to cross-compile the kernel each time. But it takes ages. If you want we should add support for kernel cross-compilation in that repo I had to do a small fix in virtme here: amluto/virtme#58 . I tried to run an emulated guest and that required qemu-system-s390x 3.1. The default qemu-syste-s390x 2.7 didn't work for me. If we need go inside the guest to run the test we need to generate correctly the rootfs for guest. Not sure if we might skip this by cross-compiling the test case for s390x and using s390x busybox binary . This approach might be useful to test other arches but of course it requires some work and maintenance. |
|
Sorry for the late reply. Good point re the s390x rootfs, I hadn't thought of that. To be honest I'd rather avoid cross compilation as much as possible. Kernels are probably OK, but doing the same for user land sounds like too much pain. Basically, we need a native CI runner. From my POV it would be acceptable if it's another big-endian arch if s390x is hard to come by. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
We are currently facing an issue using this library on s390x. I tired to run the test
TestProgramRun. I got:The text was updated successfully, but these errors were encountered: