Update the gdas.cd hash and enable GDASApp to run on WCOSS2

[RussTreadon-NOAA]

Description

This PR does the following:

1. update the sorc/gdas.cd hash to bring new GDASApp functionality into g-w
2. update env/WCOSS2.env
3. update the WCOSS2 section of ush/module-setup.sh

The change to WCOSS2.env is due to changes introduced during the fall 2024 WCOSS2 upgrade. The change to module-setup.sh is required when using spack-stack on WCOSS2.

Resolves #3219
Resolves #3100

Type of change

• Maintenance (update gdas.cd hash)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? NO
• Does this change require an update to any of the following submodules? YES
  • GDAS - this PR points at the updated sorc/gdas.cd hash. No PRs are pending.

How has this been tested?

• Clone and build on WCOSS2, Hera, Hercules, and Orion
• Run g-w CI on WCOSS2, Hera, Hercules, and Orion

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• New and existing tests pass with my changes

[RussTreadon-NOAA]

This PR is opened in draft mode until g-w CI has been run on WCOSS2, Hera, Hercules, and Orion.

The g-w team is invited to review and comment on changes to env/WCOSS2.env and ush/module-setup.sh. The changes in env/WCOSS2.env originate from the discussion in WCOSS Ticket#2024111410000051.

[aerorahul]

No issues here with the hash update.
I am not sure we are cleared to use spack-stack on WCOSS2 for regular development. The installed stack was for demonstration and testing purposes for NCO staff.

[aerorahul]

This is fine for now, but will not be acceptable for implementation. I hope there is a more robust solution than this by that time.

More importantly, this has an impact on every executable in every job -- not just GDASApp executables.

[RussTreadon-NOAA]

I completely agree. I strongly dislike these two lines. They are temporary patches to allow GFS v17 testing and development to continue on WCOSS2.

The line

export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:${HOMEgfs}/sorc/gdas.cd/build/lib"

was added because craype/2.7.17 adds

-static-libgcc -static-libstdc++ -Bstatic -lstdc++ -Bdynamic -lm -lpthread

to the ftn command. GDASApp executables failed because they could not find JEDI libraries. Might the addition of a GDASApp install option (something we must have) resolve this problem?

Another concern with the added ftn options is the following warning found in build_gdas.log

icpc: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance
ifort: warning #10315: specifying -lm before files may supersede the Intel(R) math library and affect performance

It would be unfortunate if default compiler options resulted in degraded code performance.

The line

export LD_LIBRARY_PATH="${LD_LIBRARY_PATH}:/opt/cray/pe/mpich/8.1.19/ofi/intel/19.0/lib"

was recommended by GDIT. GDASApp testing identified inconsistencies in across system modules. Some GDASApp executables failed with undefined symbol messages for mpi routines. GDIT is working on a solution.

[RussTreadon-NOAA]

g-w CI

RussTreadon-NOAA:feature/update_gdas at 4e73a31 installed on WCOSS2 (Cactus), Hera, Hercules, and Orion. The following g-w CI streams were run on each machine

• C48_ATM
• C48_S2SWA_gefs
• C48_S2SW
• C48mx500_3DVarAOWCDA
• C48mx500_hybAOWCDA
• C96C48_hybatmDA
• C96C48_hybatmaerosnowDA
• C96C48_ufs_hybatmDA
• C96_S2SWA_gefs_replay_ics
• C96_atm3DVar

All jobs in all streams successfully ran to completion on all machines except for C48mx500_hybAOWCDA job gdas_marineanlletkf on Hercules. Jobs from all other stream successfully ran to completion on Hercules. Only a single job from C48mx500_hybAOWCDA failed.

Investigation of the failure indicates a missing job dependency in the experiment xml. A rewind and rerun resulted in successful completion of the job and, as a result, the entireC48mx500_hybAOWCDA stream. @guillaumevernieres and @AndrewEichmann-NOAA have been contacted.

See issue #3219 for additional details.

[RussTreadon-NOAA]

This PR is ready for review.

[aerorahul]

Just the comments on LD_LIBRARY_PATH affecting non-gdasapp executables.
Acknowledging that gdasapp is now using space-stack on wcoss but other components are not.

[RussTreadon-NOAA]

@aerorahul , incorporating @guillaumevernieres suggestion dismissed your approval.

[RussTreadon-NOAA]

g-w issue #3222 has been opened to report the missing job dependency for C48mx500_hybAOWCDA gdas_marineanlletkf. @AndrewEichmann-NOAA will follow up on this item.

[RussTreadon-NOAA]

NCO confirmed that it is OK for GDASApp to use /apps/ops/test/spack-stack-1.6.0-nco/envs/nco-intel-19.1.3.304/install/modulefiles/Core with the understanding that it is being worked on actively for new libraries or new versions, it might change from time to time.

/apps/ops/test/spack-stack-nco/modulefiles/Core was shared as a more stable version. Tests of this version show that it does not work in GDASApp. The GDASApp build fails with

-- [bufr_query] (2.8.0)
-- Feature TESTS enabled
CMake Error at bufr-query/CMakeLists.txt:23 (find_package):
  By not providing "Findeckit.cmake" in CMAKE_MODULE_PATH this project has
  asked CMake to find a package configuration file provided by "eckit", but
  CMake did not find one.

  Could not find a package configuration file provided by "eckit" (requested
  version 1.23.0) with any of the following names:

We can not build GDASApp using /apps/dev/lmodules/core as we did in the past. Attempts to do so fail with the oops configure error

-- Adding bundle project oops
CMake Error at oops/CMakeLists.txt:14 (cmake_minimum_required):
  CMake 3.23 or higher is required.  You are running version 3.20.2


-- Configuring incomplete, errors occurred!

cmake/3.23 is not available with hpc-stack.

We will stick with /apps/ops/test/spack-stack-1.6.0-nco/envs/nco-intel-19.1.3.304/install/modulefiles/Core for the time being. This allows GFS v17 aerosol, snow, and marine DA development to continue on WCOSS2.

[RussTreadon-NOAA]

g-w CI summary

CI-Hera-Passed, CI-Orion-Passed, CI-Wcoss2-Passed labels can be applied to this PR. Tests were manually run.

CI-Hercules-Passed applies to all cases except C48mx500_hybAOWCDA. Testing discovered a missing job dependency for marine_gdasanlletkf. g-w issue #3222 has been opened to track resolution of this problem. All other g-w CI cases passed on Hercules.

[RussTreadon-NOAA]

Thank you @WalterKolczynski-NOAA