Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Bug]: tau spack test failure #228

Open
shahzebsiddiqui opened this issue Nov 7, 2023 · 0 comments
Open

[Bug]: tau spack test failure #228

shahzebsiddiqui opened this issue Nov 7, 2023 · 0 comments
Assignees
@shahzebsiddiqui
Labels
bug Something isn't working

Comments

@shahzebsiddiqui
Copy link
Contributor

CDASH Build

https://my.cdash.org/test/102913116

Link to buildspec file

https://github.com/buildtesters/buildtest-nersc/blob/devel/buildspecs/e4s/spack_test/perlmutter/23.05/tau.yml

Please describe the issue?

@wspear @sameershende

We were able to run the tau test from spack test in e4s/23.05 stack. Unfortunately there are lots of errors. It looks like its trying to run the matrix multiplication test

This one error, we could try to load the libfabric module but i think its loaded by default. I am not sure

/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/bin/ld: warning: libfabric.so.1, needed by /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libfmpich.so, not found (try using -rpath or -rpath-link)

Most of the errors are coming from undefined references, could you please take a look at this and see if we have an alternative way to test this version of tau. Perhaps write our own tau test

1 error found in test log:
     190    /global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm6
            6pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_fabric@FABRIC_1.1'
     191    /global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm6
            6pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_getinfo@FABRIC_1.3'
     192    /global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm6
            6pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_freeinfo@FABRIC_1.3'
     193    /global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm6
            6pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_version@FABRIC_1.0'

Relevant log output

_______________________________________________________________________________________________________
     The Extreme-Scale Scientific Software Stack (E4S) is accessible via the
Spack package manager.

     In order to access the production stack, you will need to load a spack
environment. Here are some tips to get started:


     'spack env list' - List all Spack environments
     'spack env activate gcc' - Activate the "gcc" Spack environment
     'spack env status' - Display the active Spack environment
     'spack load amrex' - Load the "amrex" Spack package into your user
environment

     For additional support, please refer to the following references:

       NERSC E4S Documentation: https://docs.nersc.gov/applications/e4s/
       E4S Documentation: https://e4s.readthedocs.io
       Spack Documentation: https://spack.readthedocs.io/en/latest/
       Spack Slack: https://spackpm.slack.com


______________________________________________________________________________________________________
     
==> Error: TestFailure: 3 tests failed.


Command exited with status 2:
    '/usr/bin/make' 'all'
TAU_MAKEFILE=/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib/Makefile.tau-gnu-papi-mpi-pthread-python-pdt /global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/bin/tau_cc.sh -optShared -dynamic  -g -O0  -c matmult.c -o matmult.o


Debug: Parsing with PDT Parser.
Executing> /global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/pdt-3.25.1-wzpjxzxajnj6j2qa7n3xvhphkkgmylek/craycnl/bin/cparse matmult.c -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/include -DPROFILING_ON -DTAU_PAPI -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-frkekrtdhzrwqsr2nvuntrddpju7etpt/include -DTAU_DOT_H_LESS_HEADERS -fPIC -DPTHREADS -DTAU_PTHREAD_PRELOAD -DTAU_UNIFY -DTAU_MPI -DTAU_UNIFY -DTAU_MPI_THREADED -DTAU_LINUX_TIMERS -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/python-3.10.10-q43y7fxwf5oyk225byrdvotht4qccfek/include/python3.10 -DTAU_MPIGREQUEST -DTAU_MPIDATAREP -DTAU_MPIERRHANDLER -DTAU_CRAYCNL -DTAU_AT_FORK -DTAU_MPICONSTCHAR -DTAU_MPIATTRFUNCTION -DTAU_MPITYPEEX -DTAU_MPIADDERROR -DTAU_LARGEFILE -D_LARGEFILE64_SOURCE -DTAU_BFD -DTAU_MPIFILE -DHAVE_GNU_DEMANGLE -DHAVE_TR1_HASH_MAP -DTAU_SS_ALLOC_SUPPORT -DEBS_CLOCK_RES=1 -DTAU_STRSIGNAL_OK -DTAU_UNWIND -DTAU_USE_LIBUNWIND -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libunwind-1.6.2-tnvzcrt7hriknk2nkbmed2skds5jlyyj/include -DTAU_TRACK_LD_LOADER -DTAU_USE_TLS -DTAU_MPICH3 -DTAU_OPENMPI3 -DTAU_MPI_EXTENSIONS -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/otf2-2.3-jyueqpownamjj3y43gdd24qgk4awzwo2/include -DTAU_OTF2 -DTAU_ELF_BFD -DTAU_DWARF -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libdwarf-20180129-kyqzbw724rqriqs6qtfbs6ouxebbxefd/include -I/opt/cray/pe/mpich/default/ofi/gnu/9.1/include -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/include -I/opt/cray/pe/mpich/default/ofi/gnu/9.1/include


Debug: Instrumenting with TAU
Executing> /global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/bin/tau_instrumentor matmult.pdb matmult.c -o matmult.inst.c -c


Debug: Compiling with Instrumented Code
Executing> cc -dynamic -g -O0 -c matmult.inst.c -DPROFILING_ON -DTAU_PAPI -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-frkekrtdhzrwqsr2nvuntrddpju7etpt/include -DTAU_DOT_H_LESS_HEADERS -fPIC -DPTHREADS -DTAU_PTHREAD_PRELOAD -DTAU_UNIFY -DTAU_MPI -DTAU_UNIFY -DTAU_MPI_THREADED -DTAU_LINUX_TIMERS -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/python-3.10.10-q43y7fxwf5oyk225byrdvotht4qccfek/include/python3.10 -DTAU_MPIGREQUEST -DTAU_MPIDATAREP -DTAU_MPIERRHANDLER -DTAU_CRAYCNL -DTAU_AT_FORK -DTAU_MPICONSTCHAR -DTAU_MPIATTRFUNCTION -DTAU_MPITYPEEX -DTAU_MPIADDERROR -DTAU_LARGEFILE -D_LARGEFILE64_SOURCE -DTAU_BFD -DTAU_MPIFILE -DHAVE_GNU_DEMANGLE -DHAVE_TR1_HASH_MAP -DTAU_SS_ALLOC_SUPPORT -DEBS_CLOCK_RES=1 -DTAU_STRSIGNAL_OK -DTAU_UNWIND -DTAU_USE_LIBUNWIND -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libunwind-1.6.2-tnvzcrt7hriknk2nkbmed2skds5jlyyj/include -DTAU_TRACK_LD_LOADER -DTAU_USE_TLS -DTAU_MPICH3 -DTAU_OPENMPI3 -DTAU_MPI_EXTENSIONS -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/otf2-2.3-jyueqpownamjj3y43gdd24qgk4awzwo2/include -DTAU_OTF2 -DTAU_ELF_BFD -DTAU_DWARF -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libdwarf-20180129-kyqzbw724rqriqs6qtfbs6ouxebbxefd/include -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/include -I/opt/cray/pe/mpich/default/ofi/gnu/9.1/include -o matmult.o
Looking for file: matmult.o


Debug: cleaning inst file
Executing> /bin/rm -f matmult.inst.c


Debug: cleaning PDB file
Executing> /bin/rm -f matmult.pdb

TAU_MAKEFILE=/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib/Makefile.tau-gnu-papi-mpi-pthread-python-pdt /global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/bin/tau_cc.sh -optShared -dynamic  -g -O0  -c matmult_initialize.c -o matmult_initialize.o


Debug: Parsing with PDT Parser.
Executing> /global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/pdt-3.25.1-wzpjxzxajnj6j2qa7n3xvhphkkgmylek/craycnl/bin/cparse matmult_initialize.c -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/include -DPROFILING_ON -DTAU_PAPI -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-frkekrtdhzrwqsr2nvuntrddpju7etpt/include -DTAU_DOT_H_LESS_HEADERS -fPIC -DPTHREADS -DTAU_PTHREAD_PRELOAD -DTAU_UNIFY -DTAU_MPI -DTAU_UNIFY -DTAU_MPI_THREADED -DTAU_LINUX_TIMERS -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/python-3.10.10-q43y7fxwf5oyk225byrdvotht4qccfek/include/python3.10 -DTAU_MPIGREQUEST -DTAU_MPIDATAREP -DTAU_MPIERRHANDLER -DTAU_CRAYCNL -DTAU_AT_FORK -DTAU_MPICONSTCHAR -DTAU_MPIATTRFUNCTION -DTAU_MPITYPEEX -DTAU_MPIADDERROR -DTAU_LARGEFILE -D_LARGEFILE64_SOURCE -DTAU_BFD -DTAU_MPIFILE -DHAVE_GNU_DEMANGLE -DHAVE_TR1_HASH_MAP -DTAU_SS_ALLOC_SUPPORT -DEBS_CLOCK_RES=1 -DTAU_STRSIGNAL_OK -DTAU_UNWIND -DTAU_USE_LIBUNWIND -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libunwind-1.6.2-tnvzcrt7hriknk2nkbmed2skds5jlyyj/include -DTAU_TRACK_LD_LOADER -DTAU_USE_TLS -DTAU_MPICH3 -DTAU_OPENMPI3 -DTAU_MPI_EXTENSIONS -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/otf2-2.3-jyueqpownamjj3y43gdd24qgk4awzwo2/include -DTAU_OTF2 -DTAU_ELF_BFD -DTAU_DWARF -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libdwarf-20180129-kyqzbw724rqriqs6qtfbs6ouxebbxefd/include -I/opt/cray/pe/mpich/default/ofi/gnu/9.1/include -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/include -I/opt/cray/pe/mpich/default/ofi/gnu/9.1/include


Debug: Instrumenting with TAU
Executing> /global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/bin/tau_instrumentor matmult_initialize.pdb matmult_initialize.c -o matmult_initialize.inst.c -c


Debug: Compiling with Instrumented Code
Executing> cc -dynamic -g -O0 -c matmult_initialize.inst.c -DPROFILING_ON -DTAU_PAPI -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-frkekrtdhzrwqsr2nvuntrddpju7etpt/include -DTAU_DOT_H_LESS_HEADERS -fPIC -DPTHREADS -DTAU_PTHREAD_PRELOAD -DTAU_UNIFY -DTAU_MPI -DTAU_UNIFY -DTAU_MPI_THREADED -DTAU_LINUX_TIMERS -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/python-3.10.10-q43y7fxwf5oyk225byrdvotht4qccfek/include/python3.10 -DTAU_MPIGREQUEST -DTAU_MPIDATAREP -DTAU_MPIERRHANDLER -DTAU_CRAYCNL -DTAU_AT_FORK -DTAU_MPICONSTCHAR -DTAU_MPIATTRFUNCTION -DTAU_MPITYPEEX -DTAU_MPIADDERROR -DTAU_LARGEFILE -D_LARGEFILE64_SOURCE -DTAU_BFD -DTAU_MPIFILE -DHAVE_GNU_DEMANGLE -DHAVE_TR1_HASH_MAP -DTAU_SS_ALLOC_SUPPORT -DEBS_CLOCK_RES=1 -DTAU_STRSIGNAL_OK -DTAU_UNWIND -DTAU_USE_LIBUNWIND -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libunwind-1.6.2-tnvzcrt7hriknk2nkbmed2skds5jlyyj/include -DTAU_TRACK_LD_LOADER -DTAU_USE_TLS -DTAU_MPICH3 -DTAU_OPENMPI3 -DTAU_MPI_EXTENSIONS -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/otf2-2.3-jyueqpownamjj3y43gdd24qgk4awzwo2/include -DTAU_OTF2 -DTAU_ELF_BFD -DTAU_DWARF -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libdwarf-20180129-kyqzbw724rqriqs6qtfbs6ouxebbxefd/include -I/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/include -I/opt/cray/pe/mpich/default/ofi/gnu/9.1/include -o matmult_initialize.o
Looking for file: matmult_initialize.o


Debug: cleaning inst file
Executing> /bin/rm -f matmult_initialize.inst.c


Debug: cleaning PDB file
Executing> /bin/rm -f matmult_initialize.pdb

TAU_MAKEFILE=/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib/Makefile.tau-gnu-papi-mpi-pthread-python-pdt /global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/bin/tau_cc.sh -optShared -dynamic  -g  matmult.o matmult_initialize.o -o matmult 
Debug: Moving these libraries to the end of the link line:


Debug: Linking with TAU Options
Executing> cc -dynamic -g matmult.o matmult_initialize.o -o matmult -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib/shared-gnu-papi-mpi-pthread-python-pdt -lTAU -lTAU-pthread -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib/shared-gnu-papi-mpi-pthread-python-pdt -L/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib -lfmpich -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib -lTauMpi-gnu-papi-mpi-pthread-python-pdt -lmpi -Wl,-rpath,/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-frkekrtdhzrwqsr2nvuntrddpju7etpt/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-frkekrtdhzrwqsr2nvuntrddpju7etpt/lib -lpapi -lpthread -ldl -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/lib -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/lib64 -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/lib64 -lbfd -lrt -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libunwind-1.6.2-tnvzcrt7hriknk2nkbmed2skds5jlyyj/lib -lunwind -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/python-3.10.10-q43y7fxwf5oyk225byrdvotht4qccfek/lib -lpython3.10 -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/python-3.10.10-q43y7fxwf5oyk225byrdvotht4qccfek/lib -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/otf2-2.3-jyueqpownamjj3y43gdd24qgk4awzwo2/lib -lotf2 -lotf2 -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/otf2-2.3-jyueqpownamjj3y43gdd24qgk4awzwo2/lib -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/elfutils-0.189-zskbtrajuwtvxglxyqnrplf3uved2w7t/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/elfutils-0.189-zskbtrajuwtvxglxyqnrplf3uved2w7t/lib -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libdwarf-20180129-kyqzbw724rqriqs6qtfbs6ouxebbxefd/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libdwarf-20180129-kyqzbw724rqriqs6qtfbs6ouxebbxefd/lib -ldwarf -lz -lelf -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib -lTauMpi-gnu-papi-mpi-pthread-python-pdt -L/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib -lmpi -Wl,-rpath,/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib/static-gnu-papi-mpi-pthread-python-pdt -ldl
Error: Tried looking for file: matmult
Error: Failed to link with TAU options
Error: Command(Executable) is -- cc
Error: Full Command attempted is -- cc    -dynamic  -g  matmult.o  matmult_initialize.o  -o  matmult  -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib/shared-gnu-papi-mpi-pthread-python-pdt -lTAU -lTAU-pthread         -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib/shared-gnu-papi-mpi-pthread-python-pdt -L/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib -lfmpich -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib -lTauMpi-gnu-papi-mpi-pthread-python-pdt -lmpi -Wl,-rpath,/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib         -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-frkekrtdhzrwqsr2nvuntrddpju7etpt/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-frkekrtdhzrwqsr2nvuntrddpju7etpt/lib -lpapi     -lpthread -ldl	                -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/lib -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/lib64 -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/lib64 -lbfd            -lrt -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libunwind-1.6.2-tnvzcrt7hriknk2nkbmed2skds5jlyyj/lib -lunwind           -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/python-3.10.10-q43y7fxwf5oyk225byrdvotht4qccfek/lib -lpython3.10 -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/python-3.10.10-q43y7fxwf5oyk225byrdvotht4qccfek/lib          -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/otf2-2.3-jyueqpownamjj3y43gdd24qgk4awzwo2/lib -lotf2 -lotf2 -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/otf2-2.3-jyueqpownamjj3y43gdd24qgk4awzwo2/lib   -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/elfutils-0.189-zskbtrajuwtvxglxyqnrplf3uved2w7t/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/elfutils-0.189-zskbtrajuwtvxglxyqnrplf3uved2w7t/lib -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libdwarf-20180129-kyqzbw724rqriqs6qtfbs6ouxebbxefd/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libdwarf-20180129-kyqzbw724rqriqs6qtfbs6ouxebbxefd/lib -ldwarf -lz -lelf  -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib -lTauMpi-gnu-papi-mpi-pthread-python-pdt -L/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib -lmpi -Wl,-rpath,/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib         -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib/static-gnu-papi-mpi-pthread-python-pdt     -ldl  
Error: Reverting to a Regular Make
To suppress this message and revert automatically, please add -optRevert to your TAU_OPTIONS environment variable
Press Enter to continue
 

Debug: PDT failed, switching to compiler-based instrumentation

Debug: Using compiler-based instrumentation
Debug: Moving these libraries to the end of the link line:


Debug: Linking with TAU Options
Executing> cc -dynamic -g matmult.o matmult_initialize.o -o matmult -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib/shared-gnu-papi-mpi-pthread-python-pdt -lTAU -lTAU-pthread -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib/shared-gnu-papi-mpi-pthread-python-pdt -L/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib -lfmpich -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib -lTauMpi-gnu-papi-mpi-pthread-python-pdt -lmpi -Wl,-rpath,/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-frkekrtdhzrwqsr2nvuntrddpju7etpt/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-frkekrtdhzrwqsr2nvuntrddpju7etpt/lib -lpapi -lpthread -ldl -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/lib -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/lib64 -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/lib64 -lbfd -lrt -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libunwind-1.6.2-tnvzcrt7hriknk2nkbmed2skds5jlyyj/lib -lunwind -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/python-3.10.10-q43y7fxwf5oyk225byrdvotht4qccfek/lib -lpython3.10 -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/python-3.10.10-q43y7fxwf5oyk225byrdvotht4qccfek/lib -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/otf2-2.3-jyueqpownamjj3y43gdd24qgk4awzwo2/lib -lotf2 -lotf2 -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/otf2-2.3-jyueqpownamjj3y43gdd24qgk4awzwo2/lib -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/elfutils-0.189-zskbtrajuwtvxglxyqnrplf3uved2w7t/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/elfutils-0.189-zskbtrajuwtvxglxyqnrplf3uved2w7t/lib -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libdwarf-20180129-kyqzbw724rqriqs6qtfbs6ouxebbxefd/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libdwarf-20180129-kyqzbw724rqriqs6qtfbs6ouxebbxefd/lib -ldwarf -lz -lelf -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib -lTauMpi-gnu-papi-mpi-pthread-python-pdt -L/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib -lmpi -Wl,-rpath,/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib/static-gnu-papi-mpi-pthread-python-pdt -ldl
Error: Tried looking for file: matmult
Error: Failed to link with TAU options
Error: Command(Executable) is -- cc
Error: Full Command attempted is -- cc        -dynamic    -g    matmult.o    matmult_initialize.o    -o    matmult  -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib/shared-gnu-papi-mpi-pthread-python-pdt -lTAU -lTAU-pthread         -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib/shared-gnu-papi-mpi-pthread-python-pdt -L/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib -lfmpich -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib -lTauMpi-gnu-papi-mpi-pthread-python-pdt -lmpi -Wl,-rpath,/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib         -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-frkekrtdhzrwqsr2nvuntrddpju7etpt/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/papi-6.0.0.1-frkekrtdhzrwqsr2nvuntrddpju7etpt/lib -lpapi     -lpthread -ldl	                -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/lib -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/lib64 -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/lib64 -lbfd            -lrt -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libunwind-1.6.2-tnvzcrt7hriknk2nkbmed2skds5jlyyj/lib -lunwind           -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/python-3.10.10-q43y7fxwf5oyk225byrdvotht4qccfek/lib -lpython3.10 -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/python-3.10.10-q43y7fxwf5oyk225byrdvotht4qccfek/lib          -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/otf2-2.3-jyueqpownamjj3y43gdd24qgk4awzwo2/lib -lotf2 -lotf2 -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/otf2-2.3-jyueqpownamjj3y43gdd24qgk4awzwo2/lib   -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/elfutils-0.189-zskbtrajuwtvxglxyqnrplf3uved2w7t/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/elfutils-0.189-zskbtrajuwtvxglxyqnrplf3uved2w7t/lib -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libdwarf-20180129-kyqzbw724rqriqs6qtfbs6ouxebbxefd/lib -Wl,-rpath,/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/libdwarf-20180129-kyqzbw724rqriqs6qtfbs6ouxebbxefd/lib -ldwarf -lz -lelf  -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib -lTauMpi-gnu-papi-mpi-pthread-python-pdt -L/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib -lmpi -Wl,-rpath,/opt/cray/pe/mpich/default/ofi/gnu/9.1/lib         -L/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/craycnl/lib/static-gnu-papi-mpi-pthread-python-pdt     -ldl     
 

/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/bin/ld: warning: libfabric.so.1, needed by /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libfmpich.so, not found (try using -rpath or -rpath-link)
/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_strerror@FABRIC_1.0'
/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_dupinfo@FABRIC_1.3'
/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_fabric@FABRIC_1.1'
/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_getinfo@FABRIC_1.3'
/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_freeinfo@FABRIC_1.3'
/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_version@FABRIC_1.0'
collect2: error: ld returned 1 exit status
/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/bin/ld: warning: libfabric.so.1, needed by /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libfmpich.so, not found (try using -rpath or -rpath-link)
/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_strerror@FABRIC_1.0'
/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_dupinfo@FABRIC_1.3'
/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_fabric@FABRIC_1.1'
/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_getinfo@FABRIC_1.3'
/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_freeinfo@FABRIC_1.3'
/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm66pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_version@FABRIC_1.0'
collect2: error: ld returned 1 exit status
make[1]: *** [/tmp/makefile.tau.bdtest.581988:3: all] Error 1
make: *** [Makefile:45: matmult] Error 1



1 error found in test log:
     190    /global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm6
            6pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_fabric@FABRIC_1.1'
     191    /global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm6
            6pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_getinfo@FABRIC_1.3'
     192    /global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm6
            6pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_freeinfo@FABRIC_1.3'
     193    /global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/binutils-2.40-q2sb74mwym54mqbsk7mwxx2ugvm6
            6pdc/bin/ld: /opt/cray/pe/mpich/default/ofi/gnu/9.1/lib/libmpi.so: undefined reference to `fi_version@FABRIC_1.0'
     194    collect2: error: ld returned 1 exit status
     195    make[1]: *** [/tmp/makefile.tau.bdtest.581988:3: all] Error 1
  >> 196    make: *** [Makefile:45: matmult] Error 1
     197    
     198      File "/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/bin/spack", line 54, in <module>



Command exited with status 2:
    '/usr/bin/srun' '-n' '4' 'matmult'
slurmstepd: error: execve(): matmult: No such file or directory
slurmstepd: error: execve(): matmult: No such file or directory
slurmstepd: error: execve(): matmult: No such file or directory
slurmstepd: error: execve(): matmult: No such file or directory
srun: error: nid001041: tasks 0-3: Exited with exit code 2
srun: Terminating StepId=486352.0



1 error found in test log:
     256    FAILED: Tau::test_srun_n_4_matmult: Command exited with status 2:
     257        '/usr/bin/srun' '-n' '4' 'matmult'
     258    slurmstepd: error: execve(): matmult: No such file or directory
     259    slurmstepd: error: execve(): matmult: No such file or directory
     260    slurmstepd: error: execve(): matmult: No such file or directory
     261    slurmstepd: error: execve(): matmult: No such file or directory
  >> 262    srun: error: nid001041: tasks 0-3: Exited with exit code 2
     263    srun: Terminating StepId=486352.0
     264    
     265      File "/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/bin/spack", line 54, in <module>
     266        sys.exit(main())
     267      File "/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/lib/spack/spack_installable/main.py", line 37, in main
     268        sys.exit(spack.main.main(argv))



Command exited with status 1:
    '/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg4iqb/bin/pprof'
Reading Profile files in profile.*
Error: Could not open profile.0.0.0: No such file or directory
profile.ftab: No such file or directory



1 error found in test log:
     318    Error: Could not open profile.0.0.0: No such file or directory
     319    profile.ftab: No such file or directory
     320    FAILED: Tau::test_pprof: Command exited with status 1:
     321        '/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/tau-2.32-cgk6b2wbylbpxbumi757t4yfvrwg
            4iqb/bin/pprof'
     322    Reading Profile files in profile.*
     323    Error: Could not open profile.0.0.0: No such file or directory
  >> 324    profile.ftab: No such file or directory
     325    
     326      File "/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/bin/spack", line 54, in <module>
     327        sys.exit(main())
     328      File "/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/lib/spack/spack_installable/main.py", line 37, in main
     329        sys.exit(spack.main.main(argv))
     330      File "/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/lib/spack/spack/main.py", line 1018, in main


See test log for details:
  /global/homes/b/bdtest/.spack/test/73xxed2y63q3ac3lk4djvekhikh6k7d5/tau-2.32-cgk6b2w-test-out.txt

==> Error: 1 test(s) in the suite failed.
@shahzebsiddiqui shahzebsiddiqui added the bug Something isn't working label Nov 7, 2023
@shahzebsiddiqui shahzebsiddiqui self-assigned this Nov 7, 2023
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@shahzebsiddiqui shahzebsiddiqui

Labels
bug Something isn't working
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
1 participant
@shahzebsiddiqui