Port to S4

[DavidHuber-NOAA]

Description
Ports the global workflow to the S4 cluster. Note that this does not include support for S2S experiments. Additionally, S4 does not support C768 experiments.

A couple of special notes:

• S4 does not have access to rstprod data. Among other things, this means that the nsstbufr and prepbufr.acft_profiles files must be created on the fly. The way I accomplished this was by moving the MAKE_NSSTBUFR to and creating MAKE_ACFTBUFR in config.base.emc.dyn and setting them via setup_workflow.xml. This seems like a bit of a kludge and I am open to suggestions on how else to address this. Both options need to be set for the prep and analysis jobs.
• S4 can run S2S+ experiments, but this requires significant, and convoluted, modifications to the configuration files. Support for these are thus not enabled by default. Instead, I have placed a set of configuration files in S4:/data/users/dhuber/save/s2s_configs. Users interested in performing these experiments should contact me to set it up.

Type of change

Fixes #138

Please delete options that are not relevant.

• New feature (non-breaking change which adds functionality)
• This change requires a documentation update

Suggested documentation updates

• Under global-workflow-components->data add "S4: /data/prod/glopara/dump"
• Under HPC Settings & Help
  • HPC Helpdesks - add "S4: david.huber@noaa.gov"
  • Restricted data access - add a note "NOTE: Only non-restricted data is available on S4."
  • Change the title of HPC Settings & Help -> Stacksize on Hera & Orion to "Stacksize on Hera, Orion, & S4"
  • Add a new section titled "Optimizing the global workflow on S4" with the following text
    • "The S4 cluster is relatively small and so optimizations are recommended to improve cycled runtimes. Please contact David Huber (david.huber@noaa.gov) if you are planning on running a cycled experiment on this system to obtain optimized configuration files."
  • Version - add the following row to the table:

Machine	Default	Recommended
S4	v1.8.3.1	module load git/2.30.0

• Under Clone and build global workflow
  • Quick instructions - Add [s4] to the link_workflow.sh command.
• Under Prepare initial conditions
  • Manual generation, cold starts - Prepend the note "NOTE: Initial conditions cannot be generated on S4. These must be generated on another supported platform then pushed to S4. If you do not have access to a supported system or need assistance, please contact David Huber (david.huber@noaa.gov)." References to dell and cray should be removed from this section, too.
• Under Run setup scripts to generate experiment add the following command for python on S4 module load miniconda/3.8-s4 and for rocoto module load rocoto/1.3.4.
• Under Start your run -> set up your experiment cron, remove subtitle "HPCs with access to directly edit your crontab files (WCOSS-Cray, Hera, Jet, Orion)" and the subsection "HPCs without access to directly edit your crontab files (WCOSS-Dell)"

How Has This Been Tested?

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes need updates to the documentation. I have made corresponding changes to the documentation
• My changes generate no new warnings
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

[WalterKolczynski-NOAA]

Once conflicts are resolved, I think this is ready

[DavidHuber-NOAA]

This port needs the gfs-utils repository to be ported. I'm working on that now (NOAA-EMC/gfs-utils#5).

[DavidHuber-NOAA]

I believe this branch is now ready and that all comments have been addressed.

[KateFriedman-NOAA]

@DavidHuber-NOAA FYI, another PR about to go in will change the obsproc/prepobs packages, as well as how prepobs is used (now a module). You will need to install both packages on S4 and update the module_base.s4.lua to load the prepobs module.

See #1049.

The new packages:

obsproc:
Orion install: /work/noaa/global/glopara/git/obsproc/v1.0.2
source: https://github.com/NOAA-EMC/obsproc.git
tag: OT.obsproc.v1.0.2_20220628

prepobs:
Orion install: /work/noaa/global/glopara/git/prepobs/v1.0.1
source: https://github.com/NOAA-EMC/prepobs.git
tag: OT.prepobs.v1.0.1_20220628

Note, they don't currently support S4 so you'll need to make modifications for them to build on S4.

To install (once updated for S4):

obsproc:

cd build-obsproc
INSTALL_TARGET=s4 ./build.sh

prepobs:

INSTALL_TARGET=s4 INSTALL_PREFIX=PATH_TO_TOP_OF_INSTALL ./build.sh

A new version of obsproc (v1.1.0) will be coming in with GFSv16.3 next month and develop will be updating to it soon after. Look for that update so you can install that version on S4 as well. The prepobs package will be staying at v1.0.1 with GFSv16.3.

[DavidHuber-NOAA]

@KateFriedman-NOAA Thanks for the heads up. I wasn't sure what the order would be for this, so I ported those packages and built them on S4 yesterday, so they should be good to go. I will run a test cycle today just to be sure and update module_base.s4.lua.

[KateFriedman-NOAA]

@DavidHuber-NOAA Ok cool! Final review of the obsproc PR is happening now and I hope to merge it today.

[KateFriedman-NOAA]

@DavidHuber-NOAA FYI PR #1049 (obsproc update) has just been merged into develop.

[DavidHuber-NOAA]

@KateFriedman-NOAA A cycled test completed successfully with the updated prepobs and obsproc packages on S4.

[KateFriedman-NOAA]

A cycled test completed successfully with the updated prepobs and obsproc packages on S4.

Excellent, thanks for confirming @DavidHuber-NOAA ! My approval of this PR stands. :)

[WalterKolczynski-NOAA]

@DavidHuber-NOAA Please make the needed additions to the wiki when you get the chance.

[DavidHuber-NOAA]

@WalterKolczynski-NOAA I do not have edit privileges for the Wiki. However, I did put my suggested edits in the description of this PR. Would someone be able to add those in for me or give me temporary privileges?

[DavidHuber-NOAA]

@KateFriedman-NOAA @WalterKolczynski-NOAA Would one of you be able to give me permissions to edit the wiki so I can add info on S4? If that's not possible, I have suggested edits at the top of this PR -- would you be able to add this for me? Thanks!

[KateFriedman-NOAA]

@DavidHuber-NOAA I have added your S4 updates into the global-workflow GitHub wiki. I cleaned some of the dell/cray references but not all since some (ICs section) will take more thought on another day.

[DavidHuber-NOAA]

@KateFriedman-NOAA Thank you!