Update to spack-stack 1.6.0 by DavidHuber-NOAA · Pull Request #2239 · NOAA-EMC/global-workflow

DavidHuber-NOAA · 2024-01-19T21:24:47Z

Description

This updates the module files for the global workflow to use spack-stack 1.6.0 and upgrades the submodules ~~(except GDAS)~~ to the same versions. This also addresses a couple minor issues discovered during testing: a bug in the awips_g2 job where fcsthrs was undeclared, build_upp.sh can now accept the -j # flag to specify the number of build jobs, an optimization was made to build_all.sh to build the UFS with more build jobs which speeds up the overall build process, and inconsistencies between build and run version files were corrected.

This update allows two major upgrades to take place simultaneously:

The reenabling of support for MET and METplus, so EMC_verif-global support has been reinstated and updated on all platforms, including WCOSS2.
The upgrade of CRTM to version 2.4.0.1, which now aligns with the production version to take advantage of new data sources.

With spack-stack v1.6.0, the name of the gsi-addon environment is different across platforms, so the version files have been restructured to construct the path to the Core module files.

The libraries that were upgraded in this upgrade that affect the global workflow are
crtm 2.4.0 -> 2.4.0.1
netcdf-fortran 4.6.0 -> 4.6.1
cdo 2.0.5 -> 2.2.0 (except on Orion)
prod_util 1.2.2 -> 2.1.1
py-yaml 5.4.1 -> 6.0.0
xarray 2022.3.0 -> 2023.7.0

The new libraries used in the global workflow to support EMC_verif-global are
pandas v1.5.3
python_dateutil v2.8.2
met v9.1.3
metplus v3.1.1

Resolves #2195 #1734 #1575
Depends on

Type of change

Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO

How has this been tested?

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

DavidHuber-NOAA · 2024-01-22T18:45:05Z

@malloryprow FYI I am starting to test this in cycled mode. Do you have an EMC_verif-global branch you would like me to include?

malloryprow · 2024-01-22T18:49:08Z

@malloryprow FYI I am starting to test this in cycled mode. Do you have an EMC_verif-global branch you would like me to include?

Awesome! Yes, it is https://github.com/NOAA-EMC/EMC_verif-global/tree/gw_updates, commit NOAA-EMC/EMC_verif-global@1e7baf7

DavidHuber-NOAA · 2024-01-22T19:58:07Z

@CoryMartin-NOAA @RussTreadon-NOAA Would you like to add an updated GDASApp hash that includes spack-stack 1.6.0 support to this PR? I gave it a quick try myself, but there are a couple of libraries that are not available in 1.6.0, so I'm not sure if these need to be requested or if there are appropriate substitutes:

libunistring/1.1
libidn2/2.3.4

My attempt is here: /scratch1/NCEPDEV/global/David.Huber/GW/gw_160/sorc/gdas.cd/modulefiles/GDAS/hera.lua.

RussTreadon-NOAA · 2024-01-22T20:04:02Z

It would be nice to get GDASApp to spack-stack/1.6.0 but I'm not sure if JEDI repos are ready for this. What do you think @CoryMartin-NOAA . @DavidHuber-NOAA , I will look at your hera.lua.

CoryMartin-NOAA · 2024-01-22T22:02:18Z

It would be nice to get GDASApp to spack-stack/1.6.0 but I'm not sure if JEDI repos are ready for this. What do you think @CoryMartin-NOAA . @DavidHuber-NOAA , I will look at your hera.lua.

I'm not sure. I have no reason to suspect they aren't ready but I also haven't seen JCSDA migrate to using 1.6 for Skylab yet.

RussTreadon-NOAA · 2024-01-22T22:10:10Z

While I was able to build GDASApp develop using the spack-stack/1.6.0 hera.lua, several GDASApp ctests failed

83% tests passed, 5 tests failed out of 30

Label Time Summary:
gdas-utils    =   6.47 sec*proc (9 tests)
script        =   6.47 sec*proc (9 tests)

Total Test time (real) = 113.09 sec

The following tests FAILED:
        1682 - test_gdasapp_jedi_increment_to_fv3 (Failed)
        1692 - test_gdasapp_soca_nsst_increment_to_mom6 (Failed)
        1693 - test_gdasapp_land_create_ens (Failed)
        1694 - test_gdasapp_land_imsproc (Failed)
        1696 - test_gdasapp_land_letkfoi_snowda (Failed)

Rebuild develop using spack-stack-1.5.1. Ran ctests. All tests passed.


100% tests passed, 0 tests failed out of 30

Label Time Summary:
gdas-utils    =  10.91 sec*proc (9 tests)
script        =  10.91 sec*proc (9 tests)

Total Test time (real) = 225.75 sec

RussTreadon-NOAA · 2024-01-23T19:04:36Z

The failed test_gdasapp ctests from the spack-stack/1.6.0 build were rerun with -VV. Each test has a common error message

1692:   File "/scratch1/NCEPDEV/nems/role.epic/spack-stack/spack-stack-1.6.0/envs/unified-env/install/intel/2021.5.0/py-numpy-1.22.3-fywzyg3/lib/python3.10/site-package\
s/numpy/core/overrides.py", line 6, in <module>
1692:     from numpy.core._multiarray_umath import (
1692: ModuleNotFoundError: No module named 'numpy.core._multiarray_umath'

followed by

1692: IMPORTANT: PLEASE READ THIS FOR ADVICE ON HOW TO SOLVE THIS ISSUE!
1692:
1692: Importing the numpy C-extensions failed. This error can happen for
1692: many reasons, often due to issues with your setup or how NumPy was
1692: installed.
1692:
1692: We have compiled some common reasons and troubleshooting tips at:
1692:
1692:     https://numpy.org/devdocs/user/troubleshooting-importerror.html
1692:
1692: Please note and check the following:
1692:
1692:   * The Python version is: Python3.7 from "/scratch1/NCEPDEV/da/python/opt/core/miniconda3/4.6.14/envs/gdasapp/bin/python3"
1692:   * The NumPy version is: "1.22.3"
1692:
1692: and make sure that they are the versions you expect.
1692: Please carefully study the documentation linked above for further help.
1692:
1692: Original error was: No module named 'numpy.core._multiarray_umath'

Not sure where to go with this but I'll keep poking around.

CoryMartin-NOAA · 2024-01-23T19:17:15Z

Looks like the python version used by the ctests is not the correct one. @guillaumevernieres had a similar issue with his AI tinkering on Hera. We may need to do the following in the cmake command: ``-DPYTHON_EXECUTABLE=`which python```

RussTreadon-NOAA · 2024-01-23T20:01:51Z

I made the following change to GDASApp build.sh

     module load GDAS/$BUILD_TARGET
-    CMAKE_OPTS+=" -DMPIEXEC_EXECUTABLE=$MPIEXEC_EXEC -DMPIEXEC_NUMPROC_FLAG=$MPIEXEC_NPROC -DBUILD_GSIBEC=ON"
+    CMAKE_OPTS+=" -DMPIEXEC_EXECUTABLE=$MPIEXEC_EXEC -DMPIEXEC_NUMPROC_FLAG=$MPIEXEC_NPROC -DBUILD_GSIBEC=ON -DPYTHON_EXECUTABLE=`
which python`"

Executing build.sh shows that the above expands to

Configuring ...
+ cmake -DMPIEXEC_EXECUTABLE=/apps/slurm/default/bin/srun -DMPIEXEC_NUMPROC_FLAG=-n -DBUILD_GSIBEC=ON -DPYTHON_EXECUTABLE=/scratch1/NCEPDEV/da/python/opt/core/miniconda3/4.6.14/envs/gdasapp/bin/python -DCLONE_JCSDADATA=NO -DWORKFLOW_TESTS=OFF /scratch1/NCEPDEV/da/Russ.Treadon/git/GDASApp/spack-stack/sorc

However, later on in the build I see

-- Configuring done
-- Generating done
CMake Warning:
  Manually-specified variables were not used by the project:

    PYTHON_EXECUTABLE


-- Build files have been written to: /scratch1/NCEPDEV/da/Russ.Treadon/git/GDASApp/spack-stack/build

Looks like I need to add PYTHON_EXECUTABLE to CMakeList.txt file(s) inside GDASApp. Is this the right way to go or do we upgrade the version of numpy in the gdasapp environment?

(gdasapp) Hera(hfe08):/scratch1/NCEPDEV/da/Russ.Treadon/git/GDASApp/spack-stack$ conda list numpy
# packages in environment at /scratch1/NCEPDEV/da/python/opt/core/miniconda3/4.6.14/envs/gdasapp:
#
# Name                    Version                   Build  Channel
numpy                     1.21.6                   pypi_0    pypi

CoryMartin-NOAA · 2024-01-23T21:30:38Z

Hmm, I'm pretty sure it is because the py-numpy module is loaded. Can you remove that and try again?

RussTreadon-NOAA · 2024-01-23T22:18:10Z

Unloading py-numpy/1.22.3 unloads other modules

(gdasapp) Hera(hfe08):/scratch1/NCEPDEV/da/Russ.Treadon/git/GDASApp/spack-stack/build$ module unload py-numpy/1.22.3
----------------------------------------------------------------------------------------------------------------------------------
The following dependent module(s) are not currently loaded: py-numpy/1.22.3 (required by: bufr/12.0.1), python/3.10.13 (required by: boost/1.83.0, bufr/12.0.1, fckit/0.11.0, atlas/0.35.1, py-pybind11/2.11.0)
----------------------------------------------------------------------------------------------------------------------------------

Rerun ctest_gdasapp. More tests pass. Now three tests fail

90% tests passed, 3 tests failed out of 30

Label Time Summary:
gdas-utils    =   3.34 sec*proc (9 tests)
script        =   3.34 sec*proc (9 tests)

Total Test time (real) =  21.63 sec

The following tests FAILED:
        1692 - test_gdasapp_soca_nsst_increment_to_mom6 (Failed)
        1695 - test_gdasapp_land_apply_jediincr (Failed)
        1696 - test_gdasapp_land_letkfoi_snowda (Failed)
Errors while running CTest

RussTreadon-NOAA · 2024-01-23T22:39:31Z

Forgot to add the following to my Hera environment

export OMP_NUM_THREADS=1
export SLURM_ACCOUNT=da-cpu
export SALLOC_ACCOUNT=$SLURM_ACCOUNT
export SBATCH_ACCOUNT=$SLURM_ACCOUNT
export SLURM_QOS=debug

With the above set, only one test fails

97% tests passed, 1 tests failed out of 30

Label Time Summary:
gdas-utils    =  17.31 sec*proc (9 tests)
script        =  17.31 sec*proc (9 tests)

Total Test time (real) =  72.02 sec

The following tests FAILED:
        1692 - test_gdasapp_soca_nsst_increment_to_mom6 (Failed)

I am still concerned about the other modules unloaded when py-numpy is unloaded.

DavidHuber-NOAA · 2024-01-24T13:17:42Z

@malloryprow The gfsmetp* jobs have finished running on 00z cycles from 2021110800 through 2021111500. Could you check the outputs and verify everything is as it should be? Note that there was a problem archiving 2021110900, so those .stat files are missing, but everything else should be present. The archives can be found in /scratch1/NCEPDEV/global/David.Huber/archive/metplus_data/by_VSDB under ss_160.

malloryprow · 2024-01-24T13:27:22Z

@malloryprow The gfsmetp* jobs have finished running on 00z cycles from 2021110800 through 2021111500. Could you check the outputs and verify everything is as it should be? Note that there was a problem archiving 2021110900, so those .stat files are missing, but everything else should be present. The archives can be found in /scratch1/NCEPDEV/global/David.Huber/archive/metplus_data/by_VSDB under ss_160.

How exciting to see stats being produced in the global-workflow again! The files look fine! Could I see the log files? I want to check on something.

DavidHuber-NOAA · 2024-01-24T13:29:36Z

@malloryprow Fantastic! Sure, the log files are here: /scratch1/NCEPDEV/global/David.Huber/para/comrot/ss_160/logs.

CoryMartin-NOAA · 2024-01-24T13:41:06Z

I am still concerned about the other modules unloaded when py-numpy is unloaded.

Me too, @RussTreadon-NOAA . I think this is driving me to want to replace the python env with a new one that uses spack-stack python instead of hpc-stack miniconda, but that won't be trivial. The trivial hack fix is to put in the modulefile a prepend to the PYTHONPATH var that looks in our numpy install first.

malloryprow · 2024-01-24T13:44:11Z

@malloryprow Fantastic! Sure, the log files are here: /scratch1/NCEPDEV/global/David.Huber/para/comrot/ss_160/logs.

Thanks! I was curious as where the connector script was pointing model_dir_list for EMC_verif-global to. The default value in run_verif_global_in_global_workflow.sh#L36 kicked in since model_dir is not defined in config.metp.

It would be good to get model_dir defined in config.metp. Would that be okay for this PR? It be adding one line in config.metp on line 24, like in https://github.com/malloryprow/global-workflow/blob/4acc025cdcacdba4ac85c647420a5577854deac3/parm/config/gfs/config.metp.
export model_dir=${ARCDIR}/..

DavidHuber-NOAA · 2024-01-24T13:59:28Z

@malloryprow Sure, I can put that in. I'll let it run through another cycle so you can check the logs and .stat files again.

RussTreadon-NOAA · 2024-01-24T15:54:00Z

@CoryMartin-NOAA and @DavidHuber-NOAA , I added unload("py-numpy/1.22.3") to Dave's hera.lua. I also removed the setenv(R2D2_CONFIG) line since this is no longer needed. After this I built the head of GDASApp develop and ran the ctests. All 30 tests pass.

(gdasapp) Hera(hfe10):/scratch1/NCEPDEV/da/Russ.Treadon/git/GDASApp/spack-stack/build$ ctest -R test_gdasapp
Test project /scratch1/NCEPDEV/da/Russ.Treadon/git/GDASApp/spack-stack/build
      Start 1340: test_gdasapp_util_coding_norms
 1/30 Test #1340: test_gdasapp_util_coding_norms .............   Passed    1.25 sec
      Start 1341: test_gdasapp_util_ioda_example
 2/30 Test #1341: test_gdasapp_util_ioda_example .............   Passed    7.30 sec
      Start 1342: test_gdasapp_util_prepdata
 3/30 Test #1342: test_gdasapp_util_prepdata .................   Passed    0.91 sec
      Start 1343: test_gdasapp_util_rads2ioda
 4/30 Test #1343: test_gdasapp_util_rads2ioda ................   Passed    0.40 sec
      Start 1344: test_gdasapp_util_ghrsst2ioda
 5/30 Test #1344: test_gdasapp_util_ghrsst2ioda ..............   Passed    0.14 sec
      Start 1345: test_gdasapp_util_smap2ioda
 6/30 Test #1345: test_gdasapp_util_smap2ioda ................   Passed    0.14 sec
      Start 1346: test_gdasapp_util_smos2ioda
 7/30 Test #1346: test_gdasapp_util_smos2ioda ................   Passed    0.14 sec
      Start 1347: test_gdasapp_util_viirsaod2ioda
 8/30 Test #1347: test_gdasapp_util_viirsaod2ioda ............   Passed    0.14 sec
      Start 1348: test_gdasapp_util_icecamsr2ioda
 9/30 Test #1348: test_gdasapp_util_icecamsr2ioda ............   Passed    0.16 sec
      Start 1680: test_gdasapp_check_python_norms
10/30 Test #1680: test_gdasapp_check_python_norms ............   Passed    1.36 sec
      Start 1681: test_gdasapp_check_yaml_keys
11/30 Test #1681: test_gdasapp_check_yaml_keys ...............   Passed    0.32 sec
      Start 1682: test_gdasapp_jedi_increment_to_fv3
12/30 Test #1682: test_gdasapp_jedi_increment_to_fv3 .........   Passed    1.24 sec
      Start 1683: test_gdasapp_convert_ewok_yaml
13/30 Test #1683: test_gdasapp_convert_ewok_yaml .............   Passed    0.21 sec
      Start 1684: test_gdasapp_convert_bufr_temp_dbuoy
14/30 Test #1684: test_gdasapp_convert_bufr_temp_dbuoy .......   Passed    1.12 sec
      Start 1685: test_gdasapp_convert_bufr_salt_dbuoy
15/30 Test #1685: test_gdasapp_convert_bufr_salt_dbuoy .......   Passed    0.40 sec
      Start 1686: test_gdasapp_convert_bufr_temp_mbuoyb
16/30 Test #1686: test_gdasapp_convert_bufr_temp_mbuoyb ......   Passed    0.18 sec
      Start 1687: test_gdasapp_convert_bufr_salt_mbuoyb
17/30 Test #1687: test_gdasapp_convert_bufr_salt_mbuoyb ......   Passed    0.18 sec
      Start 1688: test_gdasapp_convert_bufr_tesacprof
18/30 Test #1688: test_gdasapp_convert_bufr_tesacprof ........   Passed    0.19 sec
      Start 1689: test_gdasapp_convert_bufr_trkobprof
19/30 Test #1689: test_gdasapp_convert_bufr_trkobprof ........   Passed    0.19 sec
      Start 1690: test_gdasapp_convert_bufr_sfcships
20/30 Test #1690: test_gdasapp_convert_bufr_sfcships .........   Passed    0.19 sec
      Start 1691: test_gdasapp_convert_bufr_sfcshipsu
21/30 Test #1691: test_gdasapp_convert_bufr_sfcshipsu ........   Passed    0.19 sec
      Start 1692: test_gdasapp_soca_nsst_increment_to_mom6
22/30 Test #1692: test_gdasapp_soca_nsst_increment_to_mom6 ...   Passed    8.15 sec
      Start 1693: test_gdasapp_land_create_ens
23/30 Test #1693: test_gdasapp_land_create_ens ...............   Passed    0.83 sec
      Start 1694: test_gdasapp_land_imsproc
24/30 Test #1694: test_gdasapp_land_imsproc ..................   Passed    4.29 sec
      Start 1695: test_gdasapp_land_apply_jediincr
25/30 Test #1695: test_gdasapp_land_apply_jediincr ...........   Passed    4.96 sec
      Start 1696: test_gdasapp_land_letkfoi_snowda
26/30 Test #1696: test_gdasapp_land_letkfoi_snowda ...........   Passed   16.84 sec
      Start 1697: test_gdasapp_convert_bufr_adpsfc_snow
27/30 Test #1697: test_gdasapp_convert_bufr_adpsfc_snow ......   Passed    3.41 sec
      Start 1698: test_gdasapp_convert_bufr_adpsfc
28/30 Test #1698: test_gdasapp_convert_bufr_adpsfc ...........   Passed    4.08 sec
      Start 1699: test_gdasapp_convert_gsi_satbias
29/30 Test #1699: test_gdasapp_convert_gsi_satbias ...........   Passed    1.29 sec
      Start 1700: test_gdasapp_aero_gen_3dvar_yaml
30/30 Test #1700: test_gdasapp_aero_gen_3dvar_yaml ...........   Passed    0.30 sec

100% tests passed, 0 tests failed out of 30

Label Time Summary:
gdas-utils    =  10.57 sec*proc (9 tests)
script        =  10.57 sec*proc (9 tests)

Total Test time (real) =  61.04 sec

Not sure if this is the correct solution.

DavidHuber-NOAA · 2024-01-24T15:57:10Z

@malloryprow Alright another GFS cycle completed. Could you check the .stat and log files for the 2021111600 cycle?

CoryMartin-NOAA · 2024-01-24T16:05:49Z

@RussTreadon-NOAA if it builds, and if the 'workflow tests' pass, i.e. the ones that run the JEDI executables, then I think it might be fine.

malloryprow · 2024-01-24T16:08:52Z

@malloryprow Alright another GFS cycle completed. Could you check the .stat and log files for the 2021111600 cycle?

I see model_dir defined through config.metp in all 3 logs (gfsmetpg2g1.log, gfsmetpg2o1.log, gfsmetppcp1.log), and the stat files look good.

Since everything looks good, I would like to merge the branch these changes are in EMC_verif-global:gw_updates into EMC_verif-global:develop. I think that may need to update the branch or commit that the global-workflow is checking out? Is that right @DavidHuber-NOAA?

WalterKolczynski-NOAA · 2024-02-22T20:40:35Z

Traceback (most recent call last):
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/2239/global-workflow/scripts/exglobal_prep_land_obs.py", line 23, in <module>
    LandAnl.prepare_GTS()
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/2239/global-workflow/ush/python/wxflow/logger.py", line 266, in wrapper
    retval = func(*args, **kwargs)
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/2239/global-workflow/ush/python/pygfs/task/land_analysis.py", line 83, in prepare_GTS
    prep_gts_config = parse_j2yaml(self.task_config.GTS_OBS_LIST, localconf)
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/2239/global-workflow/ush/python/wxflow/yaml_file.py", line 176, in parse_j2yaml
    yaml_dict = YAMLFile(data=yaml_file)
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/2239/global-workflow/ush/python/wxflow/yaml_file.py", line 35, in __init__
    self.update(config)
  File "/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/2239/global-workflow/ush/python/wxflow/attrdict.py", line 120, in update
    other.update(args[0])
ValueError: dictionary update sequence element #0 has length 1; 2 is required

CoryMartin-NOAA · 2024-02-22T20:42:26Z

99.9% sure this commit would solve this issue: 8cbc974

jiaruidong2017 · 2024-02-22T20:46:07Z

Yes, I agree with Cory. This failure was caused by recent merged commit hash by changing from land to snow.

DavidHuber-NOAA · 2024-02-22T20:55:20Z

Per suggestion from @aerorahul I'm going to proceed without running the snow DA tests. We can reenable them when the issues have been sorted out in a future PR.

emcbot · 2024-02-22T20:56:44Z

CI Update on Hera at 02/22/24 08:56:43 PM
============================================
Cloning and Building global-workflow PR: 2239
with PID: 260730 on host: hfe05

jiaruidong2017 · 2024-02-22T20:58:03Z

This is caused by the following commit hash. Cory's new PR #2330 will solve this issue.

commit 0376c0812952dcdfb5ebaf3b04dbd625225a2def
Author: Cory Martin <cory.r.martin@noaa.gov>
Date:   Tue Feb 20 08:41:41 2024 -0500

    Changing 'land' to 'snow' to better reflect analysis jobs (#913)

    This PR changes 'land' everywhere to 'snow' to differentiate that this
    is snow DA and not land DA. There will need to be a corresponding G-W PR
    to do this in the workflow.

For now, you can use the early commit hashes as below for your test.

commit 49344c9084b26a6289308fc2e4433758542606b5
Author: Guillaume Vernieres <guillaume.vernieres@noaa.gov>
Date:   Sat Feb 17 09:51:18 2024 -0500

    Update to the bkg name convention (#924)

    - fixes #918

commit f09fec90b56c99ab78a1932975e15996956e6bbc
Author: Guillaume Vernieres <guillaume.vernieres@noaa.gov>
Date:   Fri Feb 16 12:59:59 2024 -0500

    Allow for gdas or gfs obs (#925)

    - fixes #921

emcbot · 2024-02-22T21:53:49Z

Automated global-workflow Testing Results:

Machine: Hera
Start: Thu Feb 22 21:06:55 UTC 2024 on hfe05
---------------------------------------------------
Build: Completed at 02/22/24 09:53:35 PM
Case setup: Skipped for experiment C48C48_ufs_hybatmDA_79d305e8
Case setup: Completed for experiment C48_ATM_79d305e8
Case setup: Completed for experiment C48_S2SW_79d305e8
Case setup: Skipped for experiment C48_S2SWA_gefs_79d305e8
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_79d305e8
Case setup: Completed for experiment C96C48_hybatmDA_79d305e8
Case setup: Completed for experiment C96_atm3DVar_79d305e8
Case setup: Skipped for experiment C96_atmsnowDA_79d305e8

emcbot · 2024-02-22T23:04:09Z

Experiment C48_ATM_79d305e8 SUCCESS on Hera at 02/22/24 11:04:08 PM

emcbot · 2024-02-22T23:48:11Z

Experiment C48_S2SW_79d305e8 SUCCESS on Hera at 02/22/24 11:48:10 PM

emcbot · 2024-02-23T00:16:19Z

Experiment C96C48_hybatmDA_79d305e8 SUCCESS on Hera at 02/23/24 12:16:19 AM

emcbot · 2024-02-23T00:16:27Z

Experiment C96_atm3DVar_79d305e8 SUCCESS on Hera at 02/23/24 12:16:26 AM

emcbot · 2024-02-23T00:20:08Z

All CI Test Cases Passed on Hera:

Experiment C48_ATM_79d305e8 *** SUCCESS *** at 02/22/24 11:04:08 PM
Experiment C48_S2SW_79d305e8 *** SUCCESS *** at 02/22/24 11:48:10 PM
Experiment C96C48_hybatmDA_79d305e8 *** SUCCESS *** at 02/23/24 12:16:19 AM
Experiment C96_atm3DVar_79d305e8 *** SUCCESS *** at 02/23/24 12:16:26 AM

Remove `FIX*` variables for fix subfolders and replace them with the remaining `FIXgfs` variable and the subfolder name (e.g. `${FIXam}` -> `${FIXgfs}/am`). The UFS_UTILS and GDASApp repos were similarly updated. This PR includes a new UFS_UTILS hash. The updated GDASAPP hash was already committed within the spack-stack/1.6.0 PR #2239. Resolves #2184

DavidHuber-NOAA added 4 commits January 19, 2024 21:11

Update modulefiles, submodules to spack-stack 1.6.0. NOAA-EMC#2195

21ff645

Corrected gsi-addon-dev spelling.

9a09d30

Corrected the SS env. name in the version files. NOAA-EMC#2195

4365f63

Merge remote-tracking branch 'origin/develop' into ss160

3ef3411

DavidHuber-NOAA added 2 commits January 22, 2024 18:56

Reenable verif-global. NOAA-EMC#2195

ccc6e2d

Better optimize build jobs (more ufs jobs).

e304bbe

DavidHuber-NOAA added 2 commits January 22, 2024 14:41

Fix gsi-addon paths for hercules, orion, and S4.

f663d47

More gsi-addon path fixes.

e400068

Disable snow DA tests

79d305e

DavidHuber-NOAA added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera and removed CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed labels Feb 22, 2024

emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Feb 22, 2024

emcbot added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Feb 22, 2024

emcbot added CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Feb 23, 2024

DavidHuber-NOAA merged commit c67393a into NOAA-EMC:develop Feb 23, 2024

DavidHuber-NOAA deleted the ss160 branch February 23, 2024 14:18

This was referenced Feb 23, 2024

Update MET+ job for COM refactor #1575

Closed

Spack-stack METplus fails with PosixPath errors #2091

Closed

DavidHuber-NOAA mentioned this pull request Feb 23, 2024

Eliminates all machine detection outside detech_machine.sh #2324

Closed

7 tasks

DavidHuber-NOAA restored the ss160 branch February 23, 2024 18:26

KateFriedman-NOAA mentioned this pull request Feb 23, 2024

Remove FIX* variables for fix subfolders #2337

Merged

7 tasks

CatherineThomas-NOAA mentioned this pull request May 31, 2024

Reenable METplus in config.base #2647

Closed

Conversation

DavidHuber-NOAA commented Jan 19, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

DavidHuber-NOAA commented Jan 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

malloryprow commented Jan 22, 2024

Uh oh!

DavidHuber-NOAA commented Jan 22, 2024

Uh oh!

RussTreadon-NOAA commented Jan 22, 2024

Uh oh!

CoryMartin-NOAA commented Jan 22, 2024

Uh oh!

RussTreadon-NOAA commented Jan 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

RussTreadon-NOAA commented Jan 23, 2024

Uh oh!

CoryMartin-NOAA commented Jan 23, 2024

Uh oh!

RussTreadon-NOAA commented Jan 23, 2024

Uh oh!

CoryMartin-NOAA commented Jan 23, 2024

Uh oh!

RussTreadon-NOAA commented Jan 23, 2024

Uh oh!

RussTreadon-NOAA commented Jan 23, 2024

Uh oh!

DavidHuber-NOAA commented Jan 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

malloryprow commented Jan 24, 2024

Uh oh!

DavidHuber-NOAA commented Jan 24, 2024

Uh oh!

CoryMartin-NOAA commented Jan 24, 2024

Uh oh!

malloryprow commented Jan 24, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidHuber-NOAA commented Jan 24, 2024

Uh oh!

RussTreadon-NOAA commented Jan 24, 2024

Uh oh!

DavidHuber-NOAA commented Jan 24, 2024

Uh oh!

CoryMartin-NOAA commented Jan 24, 2024

Uh oh!

malloryprow commented Jan 24, 2024

Uh oh!

WalterKolczynski-NOAA commented Feb 22, 2024

Uh oh!

CoryMartin-NOAA commented Feb 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jiaruidong2017 commented Feb 22, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DavidHuber-NOAA commented Feb 22, 2024

Uh oh!

emcbot commented Feb 22, 2024

Uh oh!

jiaruidong2017 commented Feb 22, 2024

Uh oh!

emcbot commented Feb 22, 2024

Uh oh!

emcbot commented Feb 22, 2024

Uh oh!

emcbot commented Feb 22, 2024

Uh oh!

emcbot commented Feb 23, 2024

Uh oh!

emcbot commented Feb 23, 2024

DavidHuber-NOAA commented Jan 19, 2024 •

edited

Loading

DavidHuber-NOAA commented Jan 22, 2024 •

edited

Loading

RussTreadon-NOAA commented Jan 22, 2024 •

edited

Loading

DavidHuber-NOAA commented Jan 24, 2024 •

edited

Loading

malloryprow commented Jan 24, 2024 •

edited

Loading

CoryMartin-NOAA commented Feb 22, 2024 •

edited

Loading

jiaruidong2017 commented Feb 22, 2024 •

edited

Loading