[Bug]: amrex spack test failing due to make target

[shahzebsiddiqui]

@etpalmer63 looping you in on this issue.

CDASH Build

https://my.cdash.org/test/102913132

Link to buildspec file

https://github.com/buildtesters/buildtest-nersc/blob/devel/buildspecs/e4s/spack_test/perlmutter/23.05/amrex.yml

Please describe the issue?

There is an issue with the makefile not having the target to run the test.

Command exited with status 2:
    '/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/gmake-4.4.1-6b72ibntxvoxfjc3vs32fsm3wlxccjgu/bin/make'
make: *** No targets specified and no makefile found.  Stop.

The test definition is declared in https://github.com/spack/spack/blob/e4s-23.05/var/spack/repos/builtin/packages/amrex/package.py#L342-L374 i can see this has changed in the develop branch.

Relevant log output

_______________________________________________________________________________________________________
     The Extreme-Scale Scientific Software Stack (E4S) is accessible via the
Spack package manager.

     In order to access the production stack, you will need to load a spack
environment. Here are some tips to get started:


     'spack env list' - List all Spack environments
     'spack env activate gcc' - Activate the "gcc" Spack environment
     'spack env status' - Display the active Spack environment
     'spack load amrex' - Load the "amrex" Spack package into your user
environment

     For additional support, please refer to the following references:

       NERSC E4S Documentation: https://docs.nersc.gov/applications/e4s/
       E4S Documentation: https://e4s.readthedocs.io
       Spack Documentation: https://spack.readthedocs.io/en/latest/
       Spack Slack: https://spackpm.slack.com


______________________________________________________________________________________________________
     
==> Error: TestFailure: 3 tests failed.


Failed to find executable '/tmp/m3503/ci/siddiq90/builds/jY_5hmbu/002/NERSC/spack-infrastructure/spack/opt/spack/__spack_path_placeholder__/__spack_path_pl/linux-sles15-zen3/gcc-11.2.0/cmake-3.26.3-j6xprhqvjwkyngr534kqdxnp5uhwf6j4/bin/cmake'




Command exited with status 2:
    '/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/gmake-4.4.1-6b72ibntxvoxfjc3vs32fsm3wlxccjgu/bin/make'
make: *** No targets specified and no makefile found.  Stop.



1 error found in test log:
     56    ==> [2023-11-07-07:43:48.258014] test: test_make: Compile
     57    ==> [2023-11-07-07:43:48.269629] Expecting return code in [0]
     58    ==> [2023-11-07-07:43:48.270198] '/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/gmake-4.4
           .1-6b72ibntxvoxfjc3vs32fsm3wlxccjgu/bin/make'
     59    make: *** No targets specified and no makefile found.  Stop.
     60    FAILED: Amrex::test_make: Command exited with status 2:
     61        '/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/opt/spack/linux-sles15-zen3/gcc-11.2.0/gmake-4.4.1-6b72ibntxvoxfjc3vs32fsm3wl
           xccjgu/bin/make'
  >> 62    make: *** No targets specified and no makefile found.  Stop.
     63    
     64      File "/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/bin/spack", line 54, in <module>
     65        sys.exit(main())
     66      File "/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/lib/spack/spack_installable/main.py", line 37, in main
     67        sys.exit(spack.main.main(argv))
     68      File "/global/common/software/spackecp/perlmutter/e4s-23.05/89639/spack/lib/spack/spack/main.py", line 1018, in main



Failed to find executable 'install_test'



See test log for details:
  /global/homes/b/bdtest/.spack/test/3uxgy6neluk6fsurma5tfufurlwmh2o2/amrex-23.05-3melsja-test-out.txt

==> Error: 1 test(s) in the suite failed.

[shahzebsiddiqui]

Do you think we could try writing our own test similar to the ones we did in past see https://github.com/buildtesters/buildtest-nersc/tree/devel/buildspecs/apps/amrex directory.

[etpalmer63]

That makes sense to me -- I can add it to my list.

[shahzebsiddiqui]

That makes sense to me -- I can add it to my list.

Thanks. In the meantime i will report this to spack so hopefully someone can fix this. I saw several other packages failing on makefile target so this might be an issue with the spack framework and not a per package issue. It hard to tell.

[etpalmer63]

I heard they recently implemented a new Spack test framework. However, I thought that would be in a more recent version. The AMReX smoke test uses an old workaround that they always knew would be temporary. So I'm curious if this could be the reason.

[shahzebsiddiqui]

@etpalmer63 i posted an issue in spack project see #224

[etpalmer63]

When I run this myself, simply changing:

module load e4s/22.11

to

module load e4s/23.05

in buildspecs/apps/amrex/anrex_single_vortex_perlmutter.yaml fixed this issue.

Ideally, I would have liked to leave it as

module load e4s

but that lead to the, ==> Error: Bad characters in version string: =11.2.0 , error.

Do you want me to submit a PR for this change? Or is there a better way to set this up so that the test doesn't need to be updated with each E4S version?

[shahzebsiddiqui]

It's good to be explicit in module load statement, the reason being system defaults will change over time. In next modulefile release e4s/23.08 will be default. New versions of software along with new compiler version can cause test divergence when you do something like this

module load e4s
spack env activate gcc
spack load amrex 
<test>

In this approach below we can guarantee test will be pinned to correct software

module load e4s/23.05
spack env activate gcc
spack load amrex 

There will be test failures due to changes in buildtest along with system CPE changes so i would like to prevent test to change dynamically overtime due to module defaults.

As per error in e4s/22.11 stack i will try to fix this today.