Add support for Ursa

[RussTreadon-NOAA]

Description

This PR contains the changes need to build and run GDASApp on Ursa

Companion PRs

None

Issues

Resolves #1695

Automated CI tests to run in Global Workflow

• atm_jjob
• C96C48_ufs_hybatmDA
• C96C48_hybatmaerosnowDA
• C48mx500_3DVarAOWCDA
• C48mx500_hybAOWCDA
• C96C48_hybatmDA

[RussTreadon-NOAA]

The changes in this PR allow GDASApp to be built and run on Ursa.

This statement comes with the following caveats

• CRTM v2.4.1-jedi.1 and CRTM v3 seg fault when built with spack-stack/1.9.1. This is the only spack-stack availale on Ursa. CRTM v3 issue #231 has been opened. Commenting out !$OMP directives over a large loop in src/CRTM_K_Matrix_Module.f90 allows CRTM v2.4.1-jedi.1 and v3 to run to completion in spack-stack/1.9.1 builds
• g-w develop does not yet support Ursa. Local changes are needed to run g-w based GDASApp ctests on Ursa. g-w issue #3674 is tracking the g-w port to Ursa.

This PR should move forward despite these caveats. Hera will be decommissioned later this year.

[DavidHuber-NOAA]

Changes look good. Just noting the increased memory requirement.

[RussTreadon-NOAA]

@DavidHuber-NOAA , thank you for your review and approval.

Details regarding test_gdasapp_atm_jjob_ens_letkf follow:

When GDASApp is built with spack-stack/1.6.0 on Hera, gdas.x requires

0: OOPS_STATS Run end                                  - Runtime:    417.12 sec,  Memory: total:    12.46 Gb, per task: min =     1.67 Gb, max =     2.35 Gb
0: Run: Finishing oops::LocalEnsembleDA<FV3JEDI, UFO and IODA observations> with status = 0

When GDASApp is built with spack-stack/1.9.1 on Ursa, gdas.x requires

0: OOPS_STATS Run end                                  - Runtime:    336.15 sec,  Memory: total:    25.18 Gb, per task: min =     3.79 Gb, max =     4.49 Gb
0: Run: Finishing oops::LocalEnsembleDA<FV3JEDI, UFO and IODA observations> with status = 0

As you note, there is a significant 2x increase in the amount of memory used with spack-stack/1.9.1.

The spack-stack/1.9.1 build use different Intel compilers than the spack-stack/1.6.0 build. Is the memory increase due to

1. different Intel compilers
2. something in spack-stack/1.9.1
3. a combination of 1 and 2
4. something else?

The --mem=96Gb added to test/atm/global-workflow/jjob_ens_letkf.sh follows the memory requirement found in g-w config.resources for this job. As the above output shows, 96Gb is not required for the GDASApp ctest. Lowering this to 28 or 32 Gb should be sufficient.

[CoryMartin-NOAA]

@guillaumevernieres @shlyaeva do you recall something like this before? Didn't we notice huge differences in SOCA memory usage on certain machines with Intel?

[shlyaeva]

@CoryMartin-NOAA yep, here's some description: #1322

[danholdaway]

Thank you Russ.

[JessicaMeixner-NOAA]

@guillaumevernieres @shlyaeva do you recall something like this before? Didn't we notice huge differences in SOCA memory usage on certain machines with Intel?

Gaea C6 requires significant more memory than WCOSS2 for running the C1152 gfs tests. You can see the memory increases here (granted they were not perfected, but just as an FYI): NOAA-EMC/global-workflow#3634

[RussTreadon-NOAA]

With caveats noted, especially the CRTM issue, I'll go ahead and merge this PR into develop.

Thank you @CoryMartin-NOAA , @DavidHuber-NOAA , and @danholdaway for your reviews and approvals.