Additional variables in diag files for WDQMS

[ADCollard]

This addresses #671. The T29 and CAT prepbufr fields are combined at output to the netCDF diag files for T, q, uv and ps as Observation_Category.

See #671 for more details.

This also includes a script for running a standalone version of the GSI.

Resolves #671

Type of change

• New feature (non-breaking change which adds functionality)

How Has This Been Tested?

Run in standalone mode. No changes to results, just additional diagnostic outputs.

Checklist

• My code follows the style guidelines of this project
• I have performed a self-review of my own code
• I have commented my code, particularly in hard-to-understand areas
• New and existing tests pass with my changes
• Any dependent changes have been merged and published

[ilianagenkova]

It's interesting to see that both CAT and T29 were already used in read_prepbufr, just not written out.
Our of curiosity, is "it29" really needed to acompany "t29"? Is that to adhere to EE2 standarts?
I am not familiar with the two *sh scripts, but will trust they are fine.

[ADCollard]

It's interesting to see that both CAT and T29 were already used in read_prepbufr, just not written out. Our of curiosity, is "it29" really needed to acompany "t29"? Is that to adhere to EE2 standarts? I am not familiar with the two *sh scripts, but will trust they are fine.

[RussTreadon-NOAA]

WCOSS2 ctests

Install ADCollard:feature/4WDQMS at a2e8157 and develop at 5f65a97 on Cactus. Run ctests (regression tests) with the following results

Test project /lfs/h2/emc/da/noscrub/russ.treadon/git/gsi/pr978/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_rdasens
    Start 4: hafs_4denvar_glbens
    Start 5: hafs_3denvar_hybens
    Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens .............   Passed  790.66 sec
2/6 Test #6: global_enkf ......................   Passed  856.40 sec
3/6 Test #2: rtma .............................   Passed  1094.46 sec
4/6 Test #5: hafs_3denvar_hybens ..............   Passed  1281.49 sec
5/6 Test #4: hafs_4denvar_glbens ..............   Passed  1460.84 sec
6/6 Test #1: global_4denvar ...................   Passed  1989.49 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) = 1989.70 sec

Ursa ctests
Repeat Cactus setup on Ursa. Run ctests with following results

Test project /scratch3/NCEPDEV/da/Russ.Treadon/git/gsi/ursa/pr978/build
    Start 1: global_4denvar
    Start 2: rtma
    Start 3: rrfs_3denvar_rdasens
    Start 4: hafs_4denvar_glbens
    Start 5: hafs_3denvar_hybens
    Start 6: global_enkf
1/6 Test #3: rrfs_3denvar_rdasens .............   Passed  487.79 sec
2/6 Test #6: global_enkf ......................   Passed  551.61 sec
3/6 Test #2: rtma .............................   Passed  728.45 sec
4/6 Test #5: hafs_3denvar_hybens ..............   Passed  852.49 sec
5/6 Test #4: hafs_4denvar_glbens ..............   Passed  912.93 sec
6/6 Test #1: global_4denvar ...................   Passed  1203.18 sec

100% tests passed, 0 tests failed out of 6

Total Test time (real) = 1203.21 sec

[RussTreadon-NOAA]

Changes look good.

None of our ctests directly check the changes in this PR. Have we verified that the resulting netcdf diagnostic files contain the correct information for various observation types?

Thank you for adding scripts to pull operational data off HPSS and run the GSI in observer mode. Two comments:

1. Developers should understand this these scripts are examples and not off-the-shelf ready to go. Developers need to update various parts of get_initial_files_ops.sh and rungsi_v17_dev.sh to fit their specific use case. They may contact us for support.
2. Will we maintain these two new scripts moving forward or leave them as frozen examples? The former requires extra diligence on our part. With the latter the scripts risk becoming obsolete as time progresses.

It's for these reasons that I have not previously committed rungsi scripts to develop.

We need to add removal of directory ush/run_observer to ush/prune_4nco_global.sh. NCO will not want to see ush/run_obsever in their para and prod installations of GSI.

[ilianagenkova]

Could @ADCollard or @RussTreadon-NOAA provide path to a sample cnv* diag file with T29/CAT added? We can use it for further WDQMS development (identifying the marine stations).

[RussTreadon-NOAA]

@ilianagenkova : Sorry, I do not have any sample files. Andrew is on leave.

[RussTreadon-NOAA]

@ADCollard , this PR can move forward when we add removal of ush/run_observer to ush/prune_4nco_global.sh. I think we can simply add run_observer to rlist on line 118 of prune_4nco_global.sh.

[ADCollard]

@RussTreadon-NOAA I changed the ush/prune_4nco_global.sh so remove the run_observer directory. I found I also had to modify it slightly as the existing script was not removing the ush/sub_* directories as intended.

[RussTreadon-NOAA]

Thank you @ADCollard for testing the prune script and fixing a bug. I don't think anyone has exercised this script in a long time.

[RussTreadon-NOAA]

Approve.

[RussTreadon-NOAA]

@ADCollard : I'll work with the GSI Review Committee to schedule merger of this PR into develop.

[ADCollard]

Thank you!