Structure and components of COMIN directory in NCO mode are modified

[chan-hoo]

DESCRIPTION OF CHANGES:

• The external model data files are stored in subdirectories 'for_ICS' and 'for_LBCS' in the nco mode (same as the community mode) and the subdirectories are removed once IC/LBC files are generated in the 'make_ics' and 'make_lbcs' tasks.
• The cycle subdirectory is added to the COMIN (COMOUT) directory by default in the nco mode.
• An intermediate symlink to 'RESTART' is added to the 'run_fcst' script for cycled runs in the nco mode.
• The 'RESTART' output directory of the 'run_fcst' task is moved to the COMIN directory only when it is not empty in the nco mode.

Type of change

• Bug fix (non-breaking change which fixes an issue)
• New feature (non-breaking change which adds functionality)
• Breaking change (fix or feature that would cause existing functionality to not work as expected)
• This change requires a documentation update

TESTS CONDUCTED:

• WE2E tests on WCOSS2:
  grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
  MET_verification
  nco_grid_RRFS_CONUS_13km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
  nco_grid_RRFS_CONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15_thompson_mynn_lam3km
  nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR

• hera.intel

• orion.intel

• cheyenne.intel

• cheyenne.gnu

• gaea.intel

• jet.intel

• wcoss2.intel

• NOAA Cloud (indicate which platform)

• Jenkins

• fundamental test suite

• comprehensive tests (specify which if a subset was used)

ISSUE:

Fixes issue mentioned in #439

CHECKLIST

• My code follows the style guidelines in the Contributor's Guide
• I have performed a self-review of my own code using the Code Reviewer's Guide
• I have commented my code, particularly in hard-to-understand areas
• My changes need updates to the documentation. I have made corresponding changes to the documentation
• My changes do not require updates to the documentation (explain).
• My changes generate no new warnings
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

[MichaelLueken]

@chan-hoo These changes look good to me! Approving.

[RatkoVasic-NOAA]

Looks good to me.

[danielabdi-noaa]

@chan-hoo Thanks, I will run a test case on Hera to verify the directory structure and maybe @MatthewPyle-NOAA too?

[danielabdi-noaa]

@chan-hoo I've run one test case on Hera and made sure that the outputs for each cycle are under own subdirectory in COMOUT, so I will approve.

[MichaelLueken]

The Jenkins CI tests have successfully completed. Merging this work now.

[danielabdi-noaa]

@chan-hoo I am having trouble getting community_ensemble_008mems running properly in NCO mode. Here are issues I am observing

• The above piece of code need to go to the bottom make_ics.sh at the very least just like in make_lbcs.sh because if make_ics fail (which is what is happening), a re-run will not fix it
• Removing the extrn_mdl_staging_dir is currently prone to race condition from different ensemble members, so if mem1 deletes it and mem2 hasn't even started running make_ics, it will fail
• Can you make it work without for_ics/lbcs appended to the staging directory and also keep the input files in COMIN just in case a re-run is needed?

[danielabdi-noaa]

Why is a symlink needed in the COMIN directory now? I think the idea in NCO mode is to keep temporary stuff in DATA so I wonder if this is avoidable.

[danielabdi-noaa]

Same questions as above.

[danielabdi-noaa]

I see from the PR description that this is for cycled runs and you did this most likely because RESTART is not accessible because it is in forecast directory which has PID attached to it. That is no more the case in PR #443 , which makes the forecast directory accessible to all other tasks by not using PID but another unique ID known to all other tasks. So with this new implementation there is no need to symlink RESTART to COMIN since the forecast directory is accessible still after restart.