WCOSS2 run support and script/config updates#1030
Merged
KateFriedman-NOAA merged 147 commits intoSep 29, 2022
Merged
Conversation
- the "EMC_post" repository was renamed to "UPP" on September 6th 2021 - update repository url in checkout script and Externals.cfg Refs: NOAA-EMC#433
- commit GFSv16.2 WCOSS2 operational GFS defs files as of April 7th 2022 - final changes to defs files will be merged later Refs: NOAA-EMC#399
- Change COMROOT/comroot back to prior path on Hera. - Add cominsyn to setup_expt.py and hosts files to set COMINsyn in config.base[.emc.dyn] on supported platforms. - When syndat is added to ops mirror to Hera change comroot to new mirror location to get syndat. Refs NOAA-EMC#419
- Change default for PACKAGEROOT and crtm_ver to ?. Refs NOAA-EMC#419
- Will make change to code makefile in new utils repo as needed. Refs NOAA-EMC#419
* develop: Fix radiance verification failing to find diag files (NOAA-EMC#1031) Supported resolutions on platforms and defaults for mode (NOAA-EMC#1026)
Contributor
Author
|
Sync merged with |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
There was a problem hiding this comment.
Found 92 potential problems in the proposed changes. Check the Files changed tab for more details.
Contributor
Author
|
@WalterKolczynski-NOAA I believe I have addressed your comments. Let me know if not or if you have any others. Thanks! |
There was a problem hiding this comment.
Found 92 potential problems in the proposed changes. Check the Files changed tab for more details.
…NOAA/global-workflow into feature/dev-wcoss2 * 'feature/dev-wcoss2' of https://github.com/KateFriedman-NOAA/global-workflow: initial commit for incoming yaml work (NOAA-EMC#1029)
There was a problem hiding this comment.
Found 92 potential problems in the proposed changes. Check the Files changed tab for more details.
- New obsproc/prepobs changes will happen in separate PR Refs NOAA-EMC#419
There was a problem hiding this comment.
Found 92 potential problems in the proposed changes. Check the Files changed tab for more details.
It was already in develop, put it back. Refs NOAA-EMC#419
There was a problem hiding this comment.
Found 92 potential problems in the proposed changes. Check the Files changed tab for more details.
…NOAA/global-workflow into feature/dev-wcoss2 * 'feature/dev-wcoss2' of https://github.com/KateFriedman-NOAA/global-workflow: Remove base_svn from Hera and Orion hosts files (NOAA-EMC#1036)
There was a problem hiding this comment.
Found 92 potential problems in the proposed changes. Check the Files changed tab for more details.
Closed
8 tasks
WalterKolczynski-NOAA
approved these changes
Sep 29, 2022
KateFriedman-NOAA
added a commit
to KateFriedman-NOAA/global-workflow
that referenced
this pull request
Jan 30, 2023
* develop: Correct issue in linking final restart files (NOAA-EMC#1285) Remove execute permissions from config files (NOAA-EMC#1281) Make needed updates to run forecast from GEFS (NOAA-EMC#1203) Remove unnecessary variables which reference to nemsio (NOAA-EMC#1259) Create analysis files for early-cycle EnKF by default (NOAA-EMC#1237) Don't wipe $DATA before running ocean bmat (NOAA-EMC#1280) More marine DA j-jobs (NOAA-EMC#1270) Update UFS-DA atmospheric prep script to be consistent with GDASApp update (NOAA-EMC#1265) Add new jjob for ocean analysis bmat (NOAA-EMC#1239) Retire ecf/versions in develop (NOAA-EMC#1267) Deploy documentation to RTD (NOAA-EMC#1264) Temporarily disable failing pytest (NOAA-EMC#1263) Remove incorrect/misleading comments in config.base (NOAA-EMC#1261) Add initial Sphinx documentation (NOAA-EMC#1258) Remove nemsio support (NOAA-EMC#1255) Increase wallclock for diag jobs (NOAA-EMC#1216) Use correct resources for GFS gempak (NOAA-EMC#1214) Abstract common j-job tasks (NOAA-EMC#1230) Add missing mkgfsawps.x link (NOAA-EMC#1218) Fix post sounding job (NOAA-EMC#1212) Revert "Use fracoro data for all new UFS applications (NOAA-EMC#1182)" (NOAA-EMC#1240) Use fracoro data for all new UFS applications (NOAA-EMC#1182) Revert "Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158)" (NOAA-EMC#1238) Add more user defined parameters for the marine DA (NOAA-EMC#1235) Update pytests action version and run sequentially (NOAA-EMC#1236) Add utility to compare Fortran namelists (NOAA-EMC#1234) Updates for pygw (NOAA-EMC#1231) Merge GFS v16.3 operational GSI changes into develop branch. (NOAA-EMC#1158) Move member up in directory hierarchy (NOAA-EMC#1201) Enable staging ics for cycled experiments. (NOAA-EMC#1199) Add tests for configuration.py (NOAA-EMC#1192) Replace ocnanal_${CDATE}} with ${RUN}ocnanal_${cyc} (NOAA-EMC#1191) define NET and RUN in the Rocoto XML to accurately mimic the ecf in ecflow (NOAA-EMC#1193) Fix checking for restart files (NOAA-EMC#1186) Fix 'DEBUG' option in build_ufs.sh (NOAA-EMC#1188) Update archive job memory request value for R&Ds (NOAA-EMC#1183) Reorder post so all flux files are generated when running offline (NOAA-EMC#1181) Stop checking for restarts on non-GFS CDUMPs (NOAA-EMC#1179) Add missing jobids in some pre-job scripts (NOAA-EMC#1176) Remove existing directory if it exists when getic runs (NOAA-EMC#1165) Add logging decorator, test and test for yaml_file (NOAA-EMC#1178) fix coding norm check in `hosts.py` (NOAA-EMC#1174) Fix some bugs and make other changes so ctest in GDASApp works (NOAA-EMC#1172) Support for the GDASApp testing in containers (NOAA-EMC#1151) ATM 3DVAR with and without IAU (NOAA-EMC#1113) Enable checking for python norms and fix violating code (NOAA-EMC#1168) Enforce decimal math in atmos post (NOAA-EMC#1171) Update marine DA j-jobs to new format (NOAA-EMC#1149) Add utility to manipulate files en masse (NOAA-EMC#1166) add action to run pytests (NOAA-EMC#1167) Pin `differential-shellcheck` to `v3` tag (NOAA-EMC#1162) Add a task base class and basic logger (NOAA-EMC#1160) Recursively convert dict to AttrDict when making an AttrDict (NOAA-EMC#1154) move configuration.py to pygw. Use it from there. return AttrDict after sourcing configs (NOAA-EMC#1153) JEDI based Marine DA tasks (NOAA-EMC#1134) Allow customizations based on user/configuration (NOAA-EMC#1146) First step towards making j-jobs consistent in use from ecflow and rocoto (NOAA-EMC#1120) enable APP=S2SWA on WCOSS2 (NOAA-EMC#1142) Fix typo in .shellcheckrc Remove prod_envir module load from WCOSS2 (NOAA-EMC#1138) Link staged GSI fix files instead of cloning them from gerrit (NOAA-EMC#1132) Address shellcheck warnings in env files (NOAA-EMC#1136) Adds group size and nmem for GEFS (NOAA-EMC#1127) Remove unnecessary sCDATE assignment in forecast_predet.sh (NOAA-EMC#1133) Convert archive jobs to proper j-jobs (NOAA-EMC#1115) Update C48 forecast to run with one thread (NOAA-EMC#1131) Improved error messages from atmos analysis (NOAA-EMC#1125) Update MODULEPATH for Orion (NOAA-EMC#1126) MPMD variable updates and fix (NOAA-EMC#1124) Introduce FHMAX_ENKF_GFS to extending ensemble forecast capabilities (NOAA-EMC#1122) Update R&D launcher commands for tasks and multi-prog (NOAA-EMC#1112) Correct crtm path in UFS DA atmospheric analysis scripts (NOAA-EMC#1111) Correct syntax in remaining sorc scripts (NOAA-EMC#1105) Add GSI background error covariance as an option for UFS DA variational assimilation (NOAA-EMC#1104) Add Early Cycle EnKF workflow (NOAA-EMC#1022) Correct errors with gdas and monitoring symlinks (NOAA-EMC#1101) Fixed gfs-utils links (NOAA-EMC#1099) Fix build scripts and bring into compliance (NOAA-EMC#1096) Feature/updates for gdas app (NOAA-EMC#1091) Change GLDAS USE_CFP to NO on Hera (NOAA-EMC#1094) Resource updates to support WCOSS2 (NOAA-EMC#1070) Set COMPILER in link for detect machine (NOAA-EMC#1092) gfs utils update (NOAA-EMC#1088) GFS-UTILS update for build and ush scripts (NOAA-EMC#1082) Update UFS version to 2022 Oct 19 (NOAA-EMC#1083) Use more cycledefs for task control (NOAA-EMC#1078) removing superfluous EFSOI-specific files from develop (NOAA-EMC#1079) Update UFS to Sept 9 version (NOAA-EMC#1073) Modify default file location for monitor data when using rocoto (NOAA-EMC#1065) Fix companion ocean resolution for C48 (NOAA-EMC#1066) Add trailing slash for gldas topo path (NOAA-EMC#1064) Limit number of CPU for post (NOAA-EMC#1061) Fix eupd trace (NOAA-EMC#1057) Port to S4 (NOAA-EMC#1023) Update to obsproc.v1.0.2 and prepobs.v1.0.1 (NOAA-EMC#1049) Add GDAS to the partial build list (NOAA-EMC#1050) Fix group number being treated as octal in gdas arch (NOAA-EMC#1053) Remove trace from link script (NOAA-EMC#1046) Update gfs-utils hash to 3a609ea (NOAA-EMC#1048) Fix link script usage statement (NOAA-EMC#1045) Replace preamble variable commands with functions (NOAA-EMC#1012) Implement fix reorg and remove gfs-utils code (NOAA-EMC#1009) Rename post scripts (NOAA-EMC#1038) Fix missing @ symbol with COMINsyn in config.base (NOAA-EMC#1039) WCOSS2 run support and script/config updates (NOAA-EMC#1030) Remove base_svn from Hera and Orion hosts files (NOAA-EMC#1036) initial commit for incoming yaml work (NOAA-EMC#1029) Fix radiance verification failing to find diag files (NOAA-EMC#1031) Supported resolutions on platforms and defaults for mode (NOAA-EMC#1026) Add GLDAS scripts & fix GLDAS job (NOAA-EMC#1018) Update GSI Monitor for radmon fix Correct shell linter config (NOAA-EMC#1013) Correct diagnostic file handling in ush/ozn_xtrct.sh (NOAA-EMC#1016) Add shell linter Github action for pull requests (NOAA-EMC#1007) Build updates for WCOSS2 (NOAA-EMC#1002) Update UFS_UTILS tag to `ufs_utils_1_8_0` (NOAA-EMC#1001) Fix preamble id (NOAA-EMC#996) Add missing "atmos" into job dependencies (NOAA-EMC#998) Bugfix in arch.sh to remove hardwired "htar" (NOAA-EMC#992) Add in stubs for aerosol DA tasks + bugfix for setup_expt where cycled and ATMA are used (NOAA-EMC#990) Add GSI monitor scripts (NOAA-EMC#969) Fix product generation at some fcst hrs (NOAA-EMC#988) Add initial config files for global aerosol DA (NOAA-EMC#986) Update diag table to remove wav-ocn coupling fields (NOAA-EMC#979) use a robust Findwgrib2.cmake to find wgrib2 built w/ native wgrib2 build (NOAA-EMC#970) Externals.cfg was stale and had drifted off (NOAA-EMC#965) Fix post comparison with zero-padded numbers (NOAA-EMC#964)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This PR includes script and run support updates for running the
developbranch on WCOSS2. This PR does not include resource updates for running on WCOSS2 and does not complete the port to WCOSS2. Additional WCOSS2developport PRs will follow this PR.Changes in this PR mostly address the following:
NWPROD/NWROOT->PACKAGEROOT) in the move of operations to WCOSS2 in the past year.WCOSS2.env, awcoss2.yamlhosts file for rocoto, and config updates.More specifically, the changes in this PR are:
WCOSS2.envunder the env folder.NWPRODtoPACKAGEROOTthroughout.NWROOTwithOPSROOTwhere appropriate.HERA.envandORION.envto add "or" for "analcalc"; needed to match similar setting inWCOSS2.envand change injobs/JGLOBAL_ATMOS_ANALYSIS_CALCjobs/JGLOBAL_ATMOS_ANALYSIS_CALCto pass "analcalc" to env file instead of "anal"compath.pyand the newcompath.pyformat; includes updatingCOMINukmetandCOMINecmwfdefaults to newcompath.pyformatCOMINobsproctojobs/JGFS_ATMOS_GEMPAK_NCDC_UPAPGIFCOMINgfswithCOMINobsprocand replaceCOMINgfs_m6hrswithCOMIN_m6hrsinjobs/JGLOBAL_ATMOS_EMCSFC_SFC_PREPARCHSYND,HOMENHCandTANK_TROPCYinjobs/JGLOBAL_ATMOS_TROPCY_QC_RELOC, removeHOMENHCp1APRUNcommands for several downstream jobs (mpirun->mpiexec)wavempexectowavempexecinstead oflauncherinjobs/JGLOBAL_WAVE_INITjobs/JGLOBAL_WAVE_POST_BNDPNTBLLto includewavepostbndpntbllinsteadwavepostbndpntPDYis invoked (. PDYto. ./PDY)RUNandRUN2variables injobs/JGFS_ATMOS_GEMPAK_PGRB2_SPECRUN_ENVIR=ncochecks and some extra whitespaces.exit $statusjobs/rocoto/vrfy.shjobs/rocoto/wafsgcip.shmodule_base.wcoss2.luato use para version ofncdiagmodule (update to use official ops install later)config.analexport nth_echgres=$nth_echgres_gfswhen CDUMP=gfs inconfig.analcalcCOMINsyntoconfig.base.emc.dynROTDIRinconfig.base.nco.staticOUTPUT_FILETYPE_variables to be "netcdf" on all machines inconfig.efcsconfig.fv3.nco.static: remove non-WCOSS2 references, addnpe_node_max=128for WCOSS2, change C384 DELTIM to 200 (ops value), and update variables for C384/C768 to ops valuesGEMPAKSH=$HOMEgfs/jobs/JGFS_ATMOS_GEMPAKline inconfig.gempak; now invoking$HOMEgfs/jobs/JGFS_ATMOS_GEMPAKdirectly in the JJOBCOMINsyndefault inconfig.prepandconfig.vrfyconfig.resources.nco.static, copy fromoperations; will need updating for GFSv17 opdconfig.vrfyRUNandRUN2variables inscripts/exgdas_atmos_nawips.sh,scripts/exgfs_atmos_goes_nawips.sh, andscripts/exgfs_atmos_nawips.shCOMINgfswithCOMINobsprocinscripts/exgfs_atmos_gempak_gif_ncdc_skew_t.shAPRUNinscripts/exgfs_atmos_gempak_meta.sh(mpirun->mpiexec)sorc/link_workflow.shto also copy the appropriate versions ofconfig.fv3andconfig.resourcesin NCO mode.NWPRODand replace with appropriate variables or paths.ush/gfs_bufr.shto print out the specific forecast hour files for SPAs to check in case of failure. (Added by GDIT and Bo Cui)ush/global_extrkr.sh(not used anywhere anymore).ftnand include-g -tracebackFFLAGSnwprodwithpackagerootin hosts files andsetup_expt.py.wcoss2.yamlhost file.workflow_tasks.pyto set native xml tag for pbspro.Issue #419
Type of change
How Has This Been Tested?
No downstream/ops-only jobs were invoked in testing, however updates to those job scripts are included in this PR.
Checklist