[Bug]: hpctoolkit spack test error

[shahzebsiddiqui]

CDASH Build

https://my.cdash.org/test/102913129

Link to buildspec file

https://github.com/buildtesters/buildtest-nersc/blob/devel/buildspecs/e4s/spack_test/perlmutter/23.05/hpctoolkit.yml

Please describe the issue?

@wyphan we are able to run the spack test for hpctoolkit for e4s/23.05 in buildtest. We noticed that spack attempted to look for g++ compiler from a hardcoded filepath from the buildcache in the testing process. This is evident in line 7 in make where its looking in /tmp/m3503.

I think this is the first time we ran spack tests for hpctoolkit. Not sure if this is still ready for use. We could still run the tests you developed in https://github.com/buildtesters/buildtest-nersc/tree/devel/buildspecs/apps/hpctoolkit

1 error found in test log:
     3    ==> [2023-11-07-07:45:16.785934] Warning: hpctoolkit: the 'test' method is deprecated. Convert stand-alone test(s) to methods with names starting 'te
          st_'.
     4    ==> [2023-11-07-07:45:16.786256] test: test: unspecified purpose
     5    ==> [2023-11-07-07:45:16.787195] 'make' '-j16' '-f' 'Makefile.spack' 'all'
     6    /tmp/m3503/ci/siddiq90/builds/PsY4QQ1Y/004/NERSC/spack-infrastructure/spack/lib/spack/env/gcc/g++ -g -O -std=c++17 -o sort sort.cpp
     7    make: /tmp/m3503/ci/siddiq90/builds/PsY4QQ1Y/004/NERSC/spack-infrastructure/spack/lib/spack/env/gcc/g++: No such file or directory
     8    make: *** [Makefile.spack:29: sort] Error 127

Relevant log output

_______________________________________________________________________________________________________
     The Extreme-Scale Scientific Software Stack (E4S) is accessible via the
Spack package manager.

     In order to access the production stack, you will need to load a spack
environment. Here are some tips to get started:


     'spack env list' - List all Spack environments
     'spack env activate gcc' - Activate the "gcc" Spack environment
     'spack env status' - Display the active Spack environment
     'spack load amrex' - Load the "amrex" Spack package into your user
environment

     For additional support, please refer to the following references:

       NERSC E4S Documentation: https://docs.nersc.gov/applications/e4s/
       E4S Documentation: https://e4s.readthedocs.io
       Spack Documentation: https://spack.readthedocs.io/en/latest/
       Spack Slack: https://spackpm.slack.com


______________________________________________________________________________________________________
     
==> Error: TestFailure: 1 test failed.


Command exited with status 2:
    'make' '-j16' '-f' 'Makefile.spack' 'all'


1 error found in test log:
     3    ==> [2023-11-07-07:45:16.785934] Warning: hpctoolkit: the 'test' method is deprecated. Convert stand-alone test(s) to methods with names starting 'te
          st_'.
     4    ==> [2023-11-07-07:45:16.786256] test: test: unspecified purpose
     5    ==> [2023-11-07-07:45:16.787195] 'make' '-j16' '-f' 'Makefile.spack' 'all'
     6    /tmp/m3503/ci/siddiq90/builds/PsY4QQ1Y/004/NERSC/spack-infrastructure/spack/lib/spack/env/gcc/g++ -g -O -std=c++17 -o sort sort.cpp
     7    make: /tmp/m3503/ci/siddiq90/builds/PsY4QQ1Y/004/NERSC/spack-infrastructure/spack/lib/spack/env/gcc/g++: No such file or directory
     8    make: *** [Makefile.spack:29: sort] Error 127
  >> 9    FAILED: Hpctoolkit::test: Command exited with status 2:
     10       'make' '-j16' '-f' 'Makefile.spack' 'all'
     11     File "/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/bin/spack", line 54, in <module>
     12       sys.exit(main())
     13     File "/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/lib/spack/spack_installable/main.py", line 37, in main
     14       sys.exit(spack.main.main(argv))
     15     File "/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/lib/spack/spack/main.py", line 1018, in main


See test log for details:
  /global/homes/b/bdtest/.spack/test/amxiiryootpdrypmypw2ncdscxctwlcw/hpctoolkit-2023.03.01-g332nvz-test-out.txt

==> Error: 1 test(s) in the suite failed.