Feature/rrfs dev2 by hu5970 · Pull Request #22 · NOAA-GSL/regional_workflow

hu5970 · 2020-12-11T20:57:35Z

DESCRIPTION OF CHANGES:

This PR is to add the following data preprocess for RRFS:

NSSL mosaic
Lightning NetCDF (NLDN obs)
METAR cloud observations
NASA LaRC cloud products
Lightning BUFR (operation obs)

This PR also adds two analysis applications for RRFS:

radar temperature tendency analysis from radar and lightning
standalone no-var cloud analysis

TESTS CONDUCTED:

The components are tested:
/mnt/lfs4/BMC/rtwbl/mhu/rrfs/dev2/RRFS_dev2

Will put those components to RRFS_dev2 soon.

ISSUE (optional):

The radar then and no-var cloud analysis are not used in the forecast step. Will link them into the cycling in next PR.

CONTRIBUTORS (optional):

Ming.Hu@noaa.gov

…dules, xml generation

… restart files.

…ation.

…, Lightning).

1. preprocess for METAR cloud 2. preprocess for NASA LaRC cloud 3. Standalone No-Var cloud analysis.

TrevorAlcott-NOAA · 2020-12-14T17:56:49Z

-        test $(find <cyclestr offset="-{{ extrn_mdl_lbcs_offset_hrs }}:00:00">{{ extrn_mdl_sysbasedir_lbcs }}/@Y@m@d@H/postprd</cyclestr> \
+        test $(find <cyclestr offset="-{{ extrn_mdl_lbcs_offset_hrs }}:00:00">{{ extrn_mdl_sysbasedir_lbcs }}/@Y@m@d@H/postprd/</cyclestr> \
               -name wrfnat_rr_[0-3][0-9].grib2 -mmin +5 | wc -l) \
        == 40


This dependency is only satisfied if the RAP forecast length is 39 hours (0 thru 39 = 40 files), or if you just happen to catch it when wrfnat_rr_xx.grib2 files have been generated for 40 of the 52 hours. This will probably lead to a lot of missed dacycles. I think you need some logic in here to look for 52 files when processing 03,09,15,21z RAP, or 22 files at all other hours, or a way to look for only the RAP files required to provide LBCs for the given dacycle or full forecast

Trevor,

I found this issue and manually changed this check to find smaller number of the RAP forecast files in dev2.
I will leave this problem to the future PR. We need to check the forecast length for boundary. We also should let this step start if certain time passed and the scripts should check older RAP cycles for the boundary.

Thanks,
Ming

…l_workflow into feature/RRFS_dev2

TrevorAlcott-NOAA · 2020-12-17T18:18:39Z

I was able to get most of the pieces in your workflow PR to run successfully in real time. A complete test would take longer just because we don’t have a recent set of “RESTART” files available due to the downtime, so the analysis step fails for the DA cycles due to missing ICs. If you can confirm you got all of this to run successfully in your own tests, I’ll approve the PR to get things moving. Can you just briefly say how you get these runs started (it seems like you need an existing dev2 system or this won't work)?

TrevorAlcott-NOAA · 2020-12-17T18:19:37Z

Also note that this PR depends on the ufs-srweather-app PR, because we need the install.sh fix in order to get the new DA executables where they need to be in /exec (NOAA-GSL/ufs-srweather-app#3)

christinaholtNOAA · 2020-12-17T18:52:19Z

@hu5970 Could you resolve the conflicts before the review? I think many of them are duplicates from the previous PR.

christinaholtNOAA

Most of the comments here are related to comments and the duplication of information. I only left a couple of comments, but please consider applying that notion to the other scripts where it makes sense.

christinaholtNOAA · 2020-12-17T21:13:39Z

+WORKDIR=${CYCLE_DIR}/novar_cldana
+if [ -r ${WORKDIR} ]; then
+  rm -fr ${WORKDIR}
+fi


This logic is not necessary (same is true in other jobs scripts).

Changed all the scripts.

christinaholtNOAA · 2020-12-17T21:15:04Z

@@ -0,0 +1,315 @@
+#!/bin/bash


The name of this script uses yet another abbreviation of "analysis". Please consider renaming it to use "anl" for clarity and consistency. I'd suggest the same for the executable name, too.

Changed from cldana to cldanl.

christinaholtNOAA · 2020-12-17T21:22:14Z

+#
+#-----------------------------------------------------------------------
+#
+# link/copy observation files to working directory 


This comment is not accurate.

christinaholtNOAA · 2020-12-17T21:35:26Z

+#-----------------------------------------------------------------------
+#
+$APRUN ./process_Lightning_nc.exe < lightning.namelist > stdout 2>&1 || print_err_msg_exit "\
+Call to executable to run radar refl process returned with nonzero exit code."


This error comment is inaccurate.

hu5970 · 2020-12-18T19:26:36Z

@TrevorAlcott-NOAA @TrevorAlcott-NOAA Just updated the code based on the comments. Please review. Thanks.

christinaholtNOAA · 2020-12-21T14:39:12Z

@@ -0,0 +1,74 @@
+MACHINE="jet"


This file should not be included in this branch.

hu5970 · 2020-12-23T17:37:54Z

@christinaholtNOAA @TrevorAlcott-NOAA Do you have further review comments on this PR?

Update dependences in FV3LAM_wflow.xml for cloud analysis.

christinaholtNOAA

@hu5970 Just a couple more comments on duplication of information. I think it looks good after this, though! :)

christinaholtNOAA · 2020-12-28T15:21:31Z

+The executable specified in EXEC does not exist:
+  EXEC = \"$EXEC\"
+Build executable and rerun."
+fi


This comment is relevant in many of the ex-scripts, and sometimes multiple times.

Since the name of the executable is left unchanged, please consider the following alternative to address the duplication of information (name of file is here multiple times):

cp_vrfy ${EXEC} ${WORKDIR}

I will make the change in the next PR.

christinaholtNOAA · 2020-12-28T15:31:07Z

+else
+   print_info_msg "$VERBOSE" "Warning: ${obs_file} does not exist!"
+fi
+


Please consider making the above section a loop to reduce the number of repeats. It's easier to maintain in the long run.

for obs_file in ${obs_files[@]} ; do if [ -r "${obs_file}" ]; then cp_vrfy "${obs_file}" $WORKDIR else print_info_msg "$VERBOSE" "Warning: ${obs_file} does not exist!" fi done

Need more change here to make it work. Most of the observation files used in DA code have different name from their files name during generation. Need to pair the observation file name with the ones used in the DA first.

hu5970 · 2020-12-29T01:37:36Z

@christinaholtNOAA Your suggested changes are good. But please merge this PR first. I have another PR now and will make your suggested change in the new PR. Thanks, Ming

…OAA-GSL#22)

Modify production generation job for real-time parallel

hu5970 added 16 commits November 17, 2020 04:37

Add data assimilation cycle to workflow, including: jobs, scripts, mo…

63dbdf5

…dules, xml generation

Add data assimilation cycle information into var_defns.sh.

bddc5dd

Add scripts to generate the FV3LAM namelist for running forecast from…

9d200b6

… restart files.

Revise scripts based on Christina's comments.

e6ae91e

Updated more scripts absed on Christina and Trevor's comments.

205d11a

Add the tasks for preprocess radar reflectivity and radar tten calcul…

eb9a922

…ation.

Update code based on Christina's review.

59e4186

Update scripts/exregional_run_analysis.sh based on Christina's comments.

52da1a7

Clean exregional_process_radarref.sh

9dd5853

Added scripts for lightning process (NetCDF).

cf98f92

Add scripts to process BUFR observation files (METAR cloud, NASA LaRC…

14493fb

…, Lightning).

Improve code based on Chiristina's comments.

2ff91bd

Add lightning data path.

5152a9a

Merge branch 'feature/RRFS_dev2_cld' into feature/RRFS_dev2

37ef087

Add no-var cloud analysis for RRFS:

fb9a46a

1. preprocess for METAR cloud 2. preprocess for NASA LaRC cloud 3. Standalone No-Var cloud analysis.

Add more accurate descriptions to the code.

932a082

hu5970 requested review from TrevorAlcott-NOAA and christinaholtNOAA December 11, 2020 20:58

TrevorAlcott-NOAA reviewed Dec 14, 2020

View reviewed changes

hu5970 added 2 commits December 16, 2020 18:18

Check if BUFR files exist before run preprocess.

13533e6

Merge branch 'feature/RRFS_dev2' of https://github.com/hu5970/regiona…

a9476e0

…l_workflow into feature/RRFS_dev2

Merge branch 'feature/RRFS_dev2' into feature/RRFS_dev2

ce5b6fb

christinaholtNOAA reviewed Dec 17, 2020

View reviewed changes

Clean the code for PR (#22) based on Christina's comments

b4e2b3d

christinaholtNOAA reviewed Dec 21, 2020

View reviewed changes

Add descriptions for namelists generated in the scripts.

c2e0542

hu5970 added 2 commits December 23, 2020 20:01

Bug fix in generate_FV3LAM_wflow.sh for generating DA workflow.

5720e07

Update dependences in FV3LAM_wflow.xml for cloud analysis.

Only give warnings if the data preprocess executable crashed.

6c7c28a

christinaholtNOAA approved these changes Dec 28, 2020

View reviewed changes

TrevorAlcott-NOAA merged commit 145796e into NOAA-GSL:feature/RRFS_dev2 Dec 31, 2020

EdwardColon-NOAA pushed a commit to EdwardColon-NOAA/regional_workflow that referenced this pull request Sep 24, 2023

Enabling compilation of "FV3_GSD_v0" CCPP suite for RRFS 13-km NA. (N…

a9e5323

…OAA-GSL#22)

guoqing-noaa pushed a commit to guoqing-noaa/regional_workflow that referenced this pull request Sep 15, 2024

Merge pull request NOAA-GSL#22 from BenjaminBlake-NOAA/feature/prdgen

332f76d

Modify production generation job for real-time parallel

Conversation

hu5970 commented Dec 11, 2020

DESCRIPTION OF CHANGES:

TESTS CONDUCTED:

ISSUE (optional):

CONTRIBUTORS (optional):

Uh oh!

TrevorAlcott-NOAA Dec 14, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

TrevorAlcott-NOAA commented Dec 17, 2020

Uh oh!

TrevorAlcott-NOAA commented Dec 17, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

christinaholtNOAA commented Dec 17, 2020

Uh oh!

christinaholtNOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

hu5970 commented Dec 18, 2020

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hu5970 commented Dec 23, 2020

Uh oh!

christinaholtNOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hu5970 Dec 29, 2020 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

hu5970 commented Dec 29, 2020

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

TrevorAlcott-NOAA Dec 14, 2020 •

edited

Loading

TrevorAlcott-NOAA commented Dec 17, 2020 •

edited

Loading

hu5970 Dec 29, 2020 •

edited

Loading