Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Compilation issue on ARM64 architecture #88

Closed
RenzoTale88 opened this issue Aug 3, 2023 · 12 comments
Closed

Compilation issue on ARM64 architecture #88

RenzoTale88 opened this issue Aug 3, 2023 · 12 comments

Comments

@RenzoTale88
Copy link

RenzoTale88 commented Aug 3, 2023

Hello,
I'm trying to compile htslib v1.17 for multiple architectures, including on an aarch64 machine with ubuntu 22.04, as part of an anaconda build. The compilation works fine on x86_64 and Mac ARM64, but fails on linux AARCH64 with the following error while compiling htscodecs:

$BUILD_PREFIX/bin/aarch64-conda-linux-gnu-cc -Wall -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/htslib-1.17 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -fvisibility=hidden  -I. -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem $PREFIX/include -c -o multipart.o multipart.c
$BUILD_PREFIX/bin/aarch64-conda-linux-gnu-cc -Wall -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/htslib-1.17 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -fvisibility=hidden  -I. -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem $PREFIX/include -c -o probaln.o probaln.c
$BUILD_PREFIX/bin/aarch64-conda-linux-gnu-cc -Wall -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/htslib-1.17 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -fvisibility=hidden  -I. -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem $PREFIX/include -c -o realn.o realn.c
$BUILD_PREFIX/bin/aarch64-conda-linux-gnu-cc -Wall -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/htslib-1.17 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -fvisibility=hidden  -I. -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem $PREFIX/include -c -o regidx.o regidx.c
$BUILD_PREFIX/bin/aarch64-conda-linux-gnu-cc -Wall -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/htslib-1.17 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -fvisibility=hidden  -I. -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem $PREFIX/include -c -o region.o region.c
$BUILD_PREFIX/bin/aarch64-conda-linux-gnu-cc -Wall -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/htslib-1.17 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -fvisibility=hidden  -I. -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem $PREFIX/include -c -o sam.o sam.c
$BUILD_PREFIX/bin/aarch64-conda-linux-gnu-cc -Wall -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/htslib-1.17 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -fvisibility=hidden  -I. -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem $PREFIX/include -c -o synced_bcf_reader.o synced_bcf_reader.c
$BUILD_PREFIX/bin/aarch64-conda-linux-gnu-cc -Wall -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/htslib-1.17 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -fvisibility=hidden  -I. -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem $PREFIX/include -c -o vcf_sweep.o vcf_sweep.c
$BUILD_PREFIX/bin/aarch64-conda-linux-gnu-cc -Wall -ftree-vectorize -fPIC -fstack-protector-strong -fno-plt -O3 -pipe -isystem $PREFIX/include -fdebug-prefix-map=$SRC_DIR=/usr/local/src/conda/htslib-1.17 -fdebug-prefix-map=$PREFIX=/usr/local/src/conda-prefix -fvisibility=hidden  -I. -DNDEBUG -D_FORTIFY_SOURCE=2 -O2 -isystem $PREFIX/include -c -o tbx.o tbx.c
cram/cram_index.c: In function 'cram_index_build':
cram/cram_index.c:743:17: warning: variable 'hpos' set but not used [-Wunused-but-set-variable]
  743 |     off_t cpos, hpos;
      |                 ^~~~
In file included from htscodecs/htscodecs/rANS_static4x16pr.c:62:
htscodecs/htscodecs/rANS_static16_int.h: In function 'encode_freq1':
htscodecs/htscodecs/rANS_static16_int.h:316:9: warning: variable 'tab_size' set but not used [-Wunused-but-set-variable]
  316 |     int tab_size = 0, i, j, z;
      |         ^~~~~~~~
htscodecs/htscodecs/rANS_static4x16pr.c: In function 'have_neon':
htscodecs/htscodecs/rANS_static4x16pr.c:1021:35: error: 'HWCAP_ASIMD' undeclared (first use in this function)
 1021 |     return (getauxval(AT_HWCAP) & HWCAP_ASIMD) != 0;
      |                                   ^~~~~~~~~~~
htscodecs/htscodecs/rANS_static4x16pr.c:1021:35: note: each undeclared identifier is reported only once for each function it appears in
htscodecs/htscodecs/rANS_static4x16pr.c:1037:1: warning: control reaches end of non-void function [-Wreturn-type]
 1037 | }
      | ^
make: *** [Makefile:184: htscodecs/htscodecs/rANS_static4x16pr.o] Error 1
Traceback (most recent call last):

I've tried also with v1.18, but with the same issue.
Any suggestion on how to fix this?
Thanks in advance
Andrea

@jkbonfield jkbonfield transferred this issue from samtools/htslib Aug 3, 2023
@jkbonfield
Copy link
Collaborator

[Transferred to htscodecs.]

What compiler is this please? Would be good to check and look at the warnings.

The HWCAP_ASIMD bit was added in 0aa64aa as a way of distinguishing which ARM CPUs have full 64-bit neon capability. It sounds like it's not always available for detection, so we'll need to think more on this.

I may be able to get an Ubuntu 22.04 AWS ARM instance running though to see some of this for myself.

@jmarshall
Copy link
Member

We did wonder in PR #63 whether some of that was going to need to be conditionalised further…

The compiler is conda-forge's build of GCC. I have checked, and the headers in the related sysroot_linux-aarch64 conda package do indeed define HWCAP_ASIMD. So this is a bit of a mystery.

As you have getauxval() and AT_HWCAP but not HWCAP_ASIMD, perhaps conda has managed to use an include/bits/hwcap.h for a different architecture. It would be instructive to add -H to your rANS_static4x16pr.o compilation command and report back what paths it has found for auxv.h, elf.h, and hwcap.h.

@RenzoTale88
Copy link
Author

@jkbonfield thanks for the quick reply! The compiler should be from anaconda's package gcc_linux-aarch64=13.1.0-hc8768d2_1:

aarch64-conda-linux-gnu-cc (conda-forge gcc 13.1.0-0) 13.1.0

Thanks for the help!
Andrea

@RenzoTale88
Copy link
Author

@jmarshall I've tried to add -H at line 488 of the Makefile:

htscodecs/htscodecs/rANS_static4x16pr.o htscodecs/htscodecs/rANS_static4x16pr.pico: htscodecs/htscodecs/rANS_static4x16pr.c config.h $(htscodecs_rANS_word_h) $(htscodecs_rANS_static4x16_h) $(htscodecs_rANS_static16_int_h) $(htscodecs_pack_h) $(htscodecs_rle_h) $(htscodecs_utils_h) $(htscodecs_rANS_static32x16pr_h) -H

But I can't see any change in the logged output. Should I apply the change somewhere else instead?

@daviesrob
Copy link
Member

daviesrob commented Aug 3, 2023

That's not the right place to do it, as that Makefile line is just a list of dependencies. It'll probably complain that it doesn't know how to make -H if it gets that far.

You could try adding -H to the CFLAGS line in config.mk. Or alternately run make to let it get as far as it can, then run it again with a slightly modified CC:

make CC='$BUILD_PREFIX/bin/aarch64-conda-linux-gnu-cc -H'

Note that you might have to substitute $BUILD_PREFIX with its value to get it to work. It should fairly quickly try to rebuild the broken source and write out the information we want.

@RenzoTale88
Copy link
Author

There you go:

. htscodecs/htscodecs/rANS_static32x16pr.h
. /tmp/tmp.qKbIz7dd2K/miniconda-lbbKSkzFXN/conda-bld/htslib_1691081142548/_build_env/aarch64-conda-linux-gnu/sysroot/usr/include/sys/auxv.h
.. /tmp/tmp.qKbIz7dd2K/miniconda-lbbKSkzFXN/conda-bld/htslib_1691081142548/_build_env/aarch64-conda-linux-gnu/sysroot/usr/include/elf.h
.. /tmp/tmp.qKbIz7dd2K/miniconda-lbbKSkzFXN/conda-bld/htslib_1691081142548/_build_env/aarch64-conda-linux-gnu/sysroot/usr/include/bits/hwcap.h
htscodecs/htscodecs/rANS_static4x16pr.c: In function 'have_neon':
htscodecs/htscodecs/rANS_static4x16pr.c:1021:35: error: 'HWCAP_ASIMD' undeclared (first use in this function)
 1021 |     return (getauxval(AT_HWCAP) & HWCAP_ASIMD) != 0;
      |                                   ^~~~~~~~~~~
htscodecs/htscodecs/rANS_static4x16pr.c:1021:35: note: each undeclared identifier is reported only once for each function it appears in
htscodecs/htscodecs/rANS_static4x16pr.c:1037:1: warning: control reaches end of non-void function [-Wreturn-type]
 1037 | }
      | ^
Multiple include guards may be useful for:
/tmp/tmp.qKbIz7dd2K/miniconda-lbbKSkzFXN/conda-bld/htslib_1691081142548/_build_env/aarch64-conda-linux-gnu/sysroot/usr/include/bits/byteswap-16.h

@daviesrob
Copy link
Member

Which version of the sysroot_linux-aarch64 conda package do you have? It looks like the latest version (sysroot_linux-aarch64-2.28-h05a177a_0) has a good hwcap.h file:

$ head -n 26 aarch64-conda-linux-gnu/sysroot/usr/include/bits/hwcap.h | tail -n 4
/* The following must match the kernel's <asm/hwcap.h>.  */
#define HWCAP_FP		(1 << 0)
#define HWCAP_ASIMD		(1 << 1)
#define HWCAP_EVTSTRM		(1 << 2)

but the previous one (sysroot_linux-aarch64-2.17-h5b4a56d_13) doesn't:

$ head -n 26 aarch64-conda-linux-gnu/sysroot/usr/include/bits/hwcap.h | tail -n 4
# error "Never include <bits/hwcap.h> directly; use <sys/auxv.h> instead."
#endif

/* No bits defined for this architecture.  */

Unfortunately I've not been able to track down why this change happened in this Conda package, but it looks like version 2.28 might fix your HTSlib compilation.

@cjw85
Copy link

cjw85 commented Aug 3, 2023

I've tweaked our CI and managed to get @RenzoTale88's build going. conda is now pulling the more recent sysroot 2.28 of its own accord. The issue seemed to be a bad conda_build_config.yaml.

https://anaconda.org/nanoporetech/htslib/files

@jmarshall
Copy link
Member

That sysroot package contains glibc headers. The aarch64 hardware capability bits were added in glibc 2.24 in 2016. Glibc 2.17 is from late 2012, only a year after aarch64 was introduced.

I suppose the code could pander to ancient libc but 🤷 maybe it's better to find out that you're not going to get SIMD…

-#elif defined(__linux__) && defined(__aarch64__)
+#elif defined(__linux__) && defined(__aarch64__) && defined(HWCAP_ASIMD)  // & similarly in other #elifs

Glad to hear your conda configuration has now been sorted out.

@cjw85
Copy link

cjw85 commented Aug 3, 2023

Conda 🙄 , I don't know why we do it to ourselves.

@jmarshall
Copy link
Member

🤣 Conda has its problems, but on a (user) population basis it's better than the alternative 🧵.

@jkbonfield
Copy link
Collaborator

jkbonfield commented Aug 4, 2023

Given there are some gcc13 warnings we should be fixing too, maybe I'll add a defined(HWCAP_ASIMD) check in there too.

Glad to hear though that this is just a broken compiler quirk which has already been resolved upstream.

Edit: Also agreed - Conda has big problems and it could have been be so much better. Too many starting points to trip up the naive first time user (which conda? which resolver? what order of evaluation in package lists? etc). However for all the conda problems we field here and my occasional gripes at it, I expect there would be even more "how do I build..." queries if it didn't exist.

jkbonfield added a commit to jkbonfield/htscodecs that referenced this issue Aug 15, 2023
A broken conda config removed HWCAP_ASIMD definition.  It's fixed now,
but we check with ifdefs as suggested by John Marshall in comments on
the issue.

Hence code is buildable, just minus NEON support on specific OS and
hardware combinations.

Fixes samtools#88
jmarshall added a commit to jmarshall/bioconda-recipes that referenced this issue Mar 5, 2024
This works around samtools/htscodecs#88, whereby the default older
sysroot_linux-aarch64 2.17 does not define HWCAP_* values on ARM and
so Neon codec implementations are unused as have_neon() is always 0.
jmarshall added a commit to jmarshall/bioconda-recipes that referenced this issue Mar 5, 2024
This works around samtools/htscodecs#88, whereby the default older
sysroot_linux-aarch64 2.17 does not define HWCAP_* values on ARM and
so Neon codec implementations are unused as have_neon() is always 0.
jmarshall added a commit to jmarshall/bioconda-recipes that referenced this issue Mar 5, 2024
This works around samtools/htscodecs#88, whereby conda-forge's older
sysroot_linux-aarch64 2.17 does not define HWCAP_* values on ARM and
so Neon codec implementations are unused as have_neon() is always 0.
Avoid this by including the kernel header to get the desired values.
jmarshall added a commit to bioconda/bioconda-recipes that referenced this issue Mar 5, 2024
* Add additional-platforms entry

* Work around old sysroot_linux-aarch64 version

This works around samtools/htscodecs#88, whereby conda-forge's older
sysroot_linux-aarch64 2.17 does not define HWCAP_* values on ARM and
so Neon codec implementations are unused as have_neon() is always 0.
Avoid this by including the kernel header to get the desired values.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

5 participants