Add rocoto sh tag, script to check netcdf file and apply this to check ocean output

[aerorahul]

Description

This PR:

• adds a Rocoto dependency tag that executes a shell command. The return code of the shell expression serves as a dependency check
• adds a script that executes ncdump on a netCDF file. If the file is a valid netCDF file, the return code is 0, else it is non-zero
• combines the above 2 to use as a dependency check for MOM6 output. If the model is still in the process of writing out the ocean output, the rocoto will execute the shell script and gather the return code.

This PR also:

• changes permissions on some ush/ scripts that did not have executable permissions.

Resolves #2328

With the addition of the shtag, the gfsocean_prod dependency will look like:

<dependency>
    <and>
        <datadep age="120"><cyclestr>&ROTDIR;/gfs.@Y@m@d/@H//model_data/ocean/history/gfs.ocean.t@Hz.6hr_avg.#dep#.nc</cyclestr></datadep>
        <sh shell="/bin/sh"><cyclestr>&HOMEgfs;/ush/check_netcdf.sh &ROTDIR;/gfs.@Y@m@d/@H//model_data/ocean/history/gfs.ocean.t@Hz.6hr_avg.#dep#.nc</cyclestr></sh>
    </and>
</dependency>

Type of change

• New feature (adds functionality)

Change characteristics

• Is this a breaking change (a change in existing functionality)? NO
• Does this change require a documentation update? YES

How has this been tested?

This was tested at low resolution and in a standalone XML where an invalid netCDF file was purposefully placed in the path to check for rocoto status.
This should be tested in the case where the issue was first seen and reported.

Checklist

• Any dependent changes have been merged and published
• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• My changes generate no new warnings
• New and existing tests pass with my changes
• I have made corresponding changes to the documentation if necessary

[NeilBarton-NOAA]

tagging @EricSinsky-NOAA for awareness

[GwenChen-NOAA]

Adding @jiandewang.

[emcbot]

Experiment C96_atmaerosnowDA failed on Hera with error logs: /scratch1/NCEPDEV/global/CI/2484/RUNTESTS/COMROOT/C96_atmaerosnowDA_96009402/logs/2021122018/gdasprepsnowobs.log

Follow link here to view the contents of the above file(s): (link)

[emcbot]

Experiment C96C48_hybatmDA failed on Hera with error logs:

/scratch1/NCEPDEV/global/CI/2484/RUNTESTS/COMROOT/C96_atmaerosnowDA_96009402/logs/2021122018/gdasprepsnowobs.log

Follow link here to view the contents of the above file(s): (link)

[emcbot]

CI FAILED Hera at 04 15 16:19:50
Built and ran in directory /scratch1/NCEPDEV/global/CI/2484

[aerorahul]

Detective @CoryMartin-NOAA traced the failure to:

str_date = re.search(r'\d{8}', self.filename).group()
        my_date = datetime.strptime(str_date, "%Y%m%d")

In the above code, a search is being performed to find 8 integers from self.filename.
self.filename in this instance is: /scratch1/NCEPDEV/stmp2/Terry.McGuinness/RUNDIRS/C96_atmaerosnowDA_96009402/gdassnowanl_18/IMSscf.20211220.C96_oro_data.nc
The first 8 integers in this filename are: 96009402. Incidentally, that is the 9600940 of the test commit. datetime is having a difficult time (pun intended) to convert 96009402 into a valid date. In the year 9600, there is no month 94 (at least on this planet).

I will merge develop and resubmit the test.

[WalterKolczynski-NOAA]

That explains it. Let me know if you need regex voodoo to find the right string.

[CoryMartin-NOAA]

I've opened an issue. It's in a JCSDA repository so it may not be fixed immediately. In the meantime, we shall hope for alphanumeric hashes only!

[aerorahul]

That explains it. Let me know if you need regex voodoo to find the right string.

@CoryMartin-NOAA has opened an issue in the JCSDA-internal/ioda-converters repo.
The new hash should not suffer from the same issue.

[CoryMartin-NOAA]

Please also note though that GDASApp will not currently build due to the RZDM meltdown.

[emcbot]

CI FAILED Hera at 04 16 16:14:40
Built and ran in directory /scratch1/NCEPDEV/global/CI/2484

[WalterKolczynski-NOAA]

@CoryMartin-NOAA This revised regex should only get an eight-digit string if it is between two periods:

str_date = re.search(r'\.(\d{8})\.', self.filename).group(1)

Test:

>>> import re
>>> test_string="abc12345678_def.87654321.xyz"
>>> print( re.search(r'\.(\d{8})\.', test_string).group(1) )                                                                                             
87654321

[emcbot]

CI Update on Wcoss2 at 04/17/24 01:18:15 AM
============================================
Cloning and Building global-workflow PR: 2484
with PID: 151480 on host: dlogin08

[emcbot]

Automated global-workflow Testing Results:

Machine: Wcoss2
Start: Wed Apr 17 01:22:48 UTC 2024 on dlogin08
---------------------------------------------------
Build: Completed at 04/17/24 01:34:02 AM
Case setup: Completed for experiment C48_ATM_00209d76
Case setup: Skipped for experiment C48mx500_3DVarAOWCDA_00209d76
Case setup: Skipped for experiment C48_S2SWA_gefs_00209d76
Case setup: Completed for experiment C48_S2SW_00209d76
Case setup: Completed for experiment C96_atm3DVar_00209d76
Case setup: Skipped for experiment C96_atmaerosnowDA_00209d76
Case setup: Completed for experiment C96C48_hybatmDA_00209d76
Case setup: Skipped for experiment C96C48_ufs_hybatmDA_00209d76

[emcbot]

Experiment C48_ATM_00209d76 SUCCESS on Wcoss2 at 04/17/24 07:12:09 AM

[emcbot]

Experiment C96_atm3DVar_00209d76 SUCCESS on Wcoss2 at 04/17/24 08:24:20 AM

[emcbot]

Experiment C96C48_hybatmDA_00209d76 SUCCESS on Wcoss2 at 04/17/24 08:24:27 AM

[emcbot]

Experiment C48_S2SW_00209d76 SUCCESS on Wcoss2 at 04/17/24 08:32:13 AM

[emcbot]

All CI Test Cases Passed on Wcoss2:

Experiment C48_ATM_00209d76 *** SUCCESS *** at 04/17/24 07:12:09 AM
Experiment C96_atm3DVar_00209d76 *** SUCCESS *** at 04/17/24 08:24:20 AM
Experiment C96C48_hybatmDA_00209d76 *** SUCCESS *** at 04/17/24 08:24:27 AM
Experiment C48_S2SW_00209d76 *** SUCCESS *** at 04/17/24 08:32:13 AM

[jiandewang]

@aerorahul my test run is successful on ocean post

addressed comments