Different results in mock and actual machine

[Aniket-Pradhan]

Hello everyone

I was trying to build python-pybv on mock, but the tests were failing for the ppc64le and the s390x architectures. Here is a mock koji scratch build on the s390x architecture. In order to replicate the test failures, we set up a GitHub workflow to test the tool on different architectures, and we found similar failures there.

To debug the issue, I signed in on the test machine for ppc64le architecture from Fedora's test machine resources and tried to build the package there. Somehow, the same SRPM was built successfully there! I went through everything again and found that this build only fails when using mock or docker (through GH Actions).

I am not sure as to why we are seeing a difference in results, even though mock is supposed to replicate a machine's architecture too, right?

The spec and the SRPMs that I tested are available here. It would be really amazing if you could help me with this issue. It's turning out to be more complicated than I first thought (I thought maybe the endianness could be an issue, but it doesn't seem like it).

I had verified if I was using the exact same SRPMs on both the builds (the actual ppc64le machine and mock). I also tested the SHA signatures of the files created in the temporary directory and they both are matching, except for a binary file. FYI, the files are created under the /tmp/pytest-of-<user>. I am guessing the way the upstream software reads the file is getting affected by the host (mock/docker) and probably that's causing in a different result as to when It is run on the test instance?

[Aniket-Pradhan]

I tried to get the CPU information on both instances (mock and the test machine for ppc64le). Below is a summary for both:

mock:

processor       : 0
vendor_id       : AuthenticAMD
cpu family      : 23
model   : 24
model name      : AMD Ryzen 5 3550H with Radeon Vega Mobile Gfx
stepping        : 1
microcode       : 0x8108102
cpu MHz         : 1390.645
cache size      : 512 KB
physical id     : 0
siblings        : 8
core id         : 0
cpu cores       : 4
apicid          : 0
initial apicid  : 0
fpu     : yes
fpu_exception   : yes
cpuid level     : 13
wp      : yes
flags   : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ht syscall nx mmxext fxsr_opt pdpe1gb rdtscp lm constant_tsc rep_good nopl nonstop_tsc cpuid extd_apicid aperfmperf pni pclmulqdq monitor ssse3 fma cx16 sse4_1 sse4_2 movbe popcnt aes xsave avx f16c rdrand lahf_lm cmp_legacy svm extapic cr8_legacy abm sse4a misalignsse 3dnowprefetch osvw skinit wdt tce topoext perfctr_core perfctr_nb bpext perfctr_llc mwaitx cpb hw_pstate ssbd ibpb vmmcall fsgsbase bmi1 avx2 smep bmi2 rdseed adx smap clflushopt sha_ni xsaveopt xsavec xgetbv1 xsaves clzero irperf xsaveerptr arat npt lbrv svm_lock nrip_save tsc_scale vmcb_clean flushbyasid decodeassists pausefilter pfthreshold avic v_vmsave_vmload vgif overflow_recov succor smca sme sev sev_es
bugs    : sysret_ss_attrs null_seg spectre_v1 spectre_v2 spec_store_bypass
bogomips        : 4192.31
TLB size        : 2560 4K pages
clflush size    : 64
cache_alignment : 64
address sizes   : 43 bits physical, 48 bits virtual
power management: ts ttp tm hwpstate eff_freq_ro [13] [14]
...

Test machine instance:

processor       : 0
cpu             : POWER9 (architected), altivec supported
clock           : 2250.000000MHz
revision        : 2.2 (pvr 004e 1202)

processor       : 1
cpu             : POWER9 (architected), altivec supported
clock           : 2250.000000MHz
revision        : 2.2 (pvr 004e 1202)

processor       : 2
cpu             : POWER9 (architected), altivec supported
clock           : 2250.000000MHz
revision        : 2.2 (pvr 004e 1202)

processor       : 3
cpu             : POWER9 (architected), altivec supported
clock           : 2250.000000MHz
revision        : 2.2 (pvr 004e 1202)

timebase        : 512000000
platform        : pSeries
model           : IBM pSeries (emulated by qemu)
machine         : CHRP IBM pSeries (emulated by qemu)
MMU             : Radix

This shouldn't particularly affect the upstream code in my opinion, but this was the only difference I could find between the two instances.

[xsuchy]

Where do you build ppc64le and s390x builds using Mock? Is it in Copr? Or on your machine? On which architecture?
Because it seems that you are doing the builds using emulation https://github.com/rpm-software-management/mock/wiki/Feature-forcearch The emulated architecture usually works, but there are known issues. These issues have to be addressed with QEMU project.

[Aniket-Pradhan]

I am trying to run the builds on my laptop, with an AMD Ryzen CPU - x86_64 architecture.

The emulated architecture doesn't seem to be working for me, as you pointed out. Seems to be an issue with qemu-user-static which only emulates the user-space and not the kernel-space. We were facing the same issue when running the software on docker with fedora-ppc64le.

Probably asking at the wrong place again, but, Is there any other way we can use koji//mock to submit builds since we have verified that the tool works fine on a native VM but fails because of being run on a different architecture when using an emulator?

[xsuchy]

Copr use emulated ppc64le and s390x. But Koji uses native architecture.
Or if it is Fedora related you can use https://fedoraproject.org/wiki/Test_Machine_Resources_For_Package_Maintainers

The point of user space vs. kernel space is relevant I put this as a warning on https://github.com/rpm-software-management/mock/wiki/Feature-forcearch page.