Modify hashes of external repos to obtain a combination that works to…

[gsketefian]

DESCRIPTION OF CHANGES:

• Modify hashes of external repos to obtain a combination that works together (builds + runs the workflow successfully E2E). The hashes and their creation dates are as follows:

regional_workflow  f371ada7    Nov. 05, 2020
UFS_UTILS          bf705493    Oct. 20, 2020
ufs-weather-model  718c1ee4    Oct. 07, 2020
EMC_post           2689a70c    Oct. 20, 2020

Some of these are somewhat old, but using these is better than having a broken workflow. We hope PR #62 (and possibly others) will fix the cmake build and allow more recent versions of the above repos to be used. This PR is a stopgap until that happens.

• Add the two new UFS_UTILS executables fvcom_to_FV3 and vcoord_gen to install_all.sh so that they get copied to the bin directory.

TESTS CONDUCTED:

• Built successfully on Hera.
• Ran one WE2E test (DOT_OR_USCORE) successfully on Hera.

ISSUE (optional):

• Addresses Issue Master inconsistent with regional_workflow/develop #57.

[gsketefian]

It would be great to get this tested on Hera, Jet, and Cheyenne ASAP.

IMPORTANT: You must use the "build_all.sh" script to build the codes. The newer "cmake" approach doesn't work with this combination of hashes.

Thx.

[mkavulich]

Testing on Cheyenne

[JeffBeck-NOAA]

The software stack has changed on Jet since this hash of UFS_UTILS. The included build.jet modulefile doesn't work, but this combination does:

module use /contrib/sutils/modulefiles
module load sutils
module load intel/18.0.5.274
module load impi/2018.4.274
module load hdf5/1.10.5
module load cmake/3.16.1

module use /lfs4/HFIP/hfv3gfs/software/NCEPLIBS-ufs-v2.0.0/intel-18.0.5.274/impi-2018.4.274/modules

module load NCEPLIBS/2.0.0
module load esmf/8.0.0

[christinaholtNOAA]

How would you suggest that difference in software stack is managed here?

[mkavulich]

This hash of regional_workflow will not work on Cheyenne (or any other platform without "partition_default", "partition_hpss", or "partition_fcst" settings) due to commit fb8f539 which breaks the workflow generation python script. Regional_workflow will need to be updated with the same changes as #343 in order to work on Cheyenne.

I would recommend not letting that hold up this PR, since at least this will work on some platforms, and fixing this for Cheyenne will require a new PR to regional_workflow. I will work on opening that ASAP.

[christinaholtNOAA]

@gsketefian I'm just realizing that the hash you are proposing from the model is still from the release branch. Is there any intent to move back to the develop branch? Pointing to some hash of the develop branches of the externals is important to have a method to test developments to more than just the workflow.

[gsketefian]

@christinaholtNOAA Yes, that's right. I chose one that has many/most of the suites in there so it would be useful to you. Is there a specific feature(s) of the develop branch you're looking for? I can look for a hash from the develop branch that has that feature(s). The EMC_post hash is also from the release branch. Do you need that changed to a develop hash as well?
@mkavulich is working on getting the build to work with the current HEADs of all the externals, after which we plan to freeze the hashes.

[christinaholtNOAA]

All I need currently is for the top of master to work so that I can get some PRs through. My question was a more general question about the management of the externals.

[JeffBeck-NOAA]

@gsketefian, I'd recommend we close this PR, since we're moving toward using the HEADs of all repos.

[christinaholtNOAA]

@JeffBeck-NOAA Are you using the heads, or hashes near the top of the main branches? Using the branches themselves will likely get us into a situation again in the future where it's difficult to find a set of repos that work together.

[climbfuji]

Agreed. Please always point to fixed hashes.

…

On Dec 9, 2020, at 11:53 AM, Christina Holt ***@***.***> wrote: @JeffBeck-NOAA <https://github.com/JeffBeck-NOAA> Are you using the heads, or hashes near the top of the main branches? Using the branches themselves will likely get us into a situation again in the future where it's difficult to find a set of repos that work together. — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#63 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB5C2RIPOIPDNBKHYRYD3Y3ST7BTRANCNFSM4UH5RFFQ>.

[gsketefian]

@christinaholtNOAA @JeffBeck-NOAA My suggestion is to always use hashes and then update them in a controlled manner.

[gsketefian]

Closing this PR.

[JeffBeck-NOAA]

@JeffBeck-NOAA Are you using the heads, or hashes near the top of the main branches? Using the branches themselves will likely get us into a situation again in the future where it's difficult to find a set of repos that work together.

@christinaholtNOAA, yep, definitely meant hashes near the HEADs. In addition, we will eventually create a "stable" branch in regional_workflow and ufs-srweather-app that will point to thoroughly tested hashes to avoid this problem in the future.