Adding large-scale blending step to regional_workflow. by delippi · Pull Request #540 · NOAA-GSL/regional_workflow

delippi · 2023-06-28T19:48:45Z

DESCRIPTION OF CHANGES:

Two python scripts were added to perform the blending and pre-blending steps: da_blending_fv3.py and da_chgres_cold2fv3.py. These python scripts perform the blending and pre-blending steps respectively by calling chgres_winds, remap_dwinds, and chgres_winds *.so files which should be compiled using f2py (see NOAA-GSL/rrfs_utl#38) such that they can be imported as python modules. The python scripts fit into the exregional_make_ics.sh script where all the necessary files are also linked or copied into the appropriate locations. Some default values for DO_BLENDING etc were also updated.

TESTS CONDUCTED:

Timing: Total added run time is about 90s. This is possible since the python scripts make use of the f2py modules and the netcdf4 library. The python is just used as glue and doesn't perform any expensive calculations.

Have tested these scripts in the context of a retro to make sure everything is called correctly, but there has not yet been any scientific evaluation. There have also not been any forecasts run from the newly blended files.

DEPENDENCIES:

Add blending and associated pre-processing tools to rrfs_utl:
NOAA-GSL/rrfs_utl#38

DOCUMENTATION:

BlendingGlobal_to_Regional_Proposal ppt: https://docs.google.com/presentation/d/1AGXqGar1rNtKOXXhoTTD_tZ6nFg8LeZQ/edit#slide=id.p1

Large Scale Blending Tests ppt:
https://docs.google.com/presentation/d/1dVN3ZphUlVv8EIpf9Tev3ZciTMHi3oUHbNUNBJjayKE/edit?pli=1#slide=id.g21875f2a523_0_15

Simplified schematic of how the blending steps fit into the workflow:

JacobCarley-NOAA · 2023-06-28T20:01:34Z

+    v_w = np.float64(coldnc["v_w"][:, :, :])   # (128, 768, 769)
+
+    # grid data - usually has 2x as many grid points, so we need every other value.
+    gridx = np.float64(gridnc["x"][0:-1:2, 0:-1:2])


do you know why this has 2x as many points? is this the super grid?

No, I'm not sure why it has 2x as many. For example:

ncdump -h /lfs/h2/emc/global/noscrub/emc.global/FIX/fix/orog/20220805/C768/C768_grid.tile1.nc netcdf C768_grid.tile1 { dimensions: string = 255 ; nx = 1536 ; ny = 1536 ; nxp = 1537 ; nyp = 1537 ;

I just know that this part is related only to the wind rotation part which I was able to reproduce within rounding error.

guoqing-noaa · 2023-06-30T23:13:20Z

 # is because the radiance DA in EnKF relies the radiance procedures in the GSI-observer, 
 # which is mainly controled by DO_RADDA.
+#
+# DO_BLENDING:


suggest changing DO_BLENDING to "DO_ENS_BLENDING" to make it more clear.
It is possible that we may do blending for the deterministic part in the future.

guoqing-noaa · 2023-07-04T22:36:06Z

+#echo first_cycle date         : ${DATE_FIRST_CYCL}${STARTHOUR}
+#echo first blended cycle date : $FIRST_BLENDED_CYCLE_DATE
+if [[ $cdate_crnt_fhr -ge ${FIRST_BLENDED_CYCLE_DATE} ]]; then
+if [[ $DO_BLENDING == "TRUE" && $EXTRN_MDL_NAME_ICS == "GDASENKF" ]]; then


Logically, lines 721-726 is expected to be put inside the if DO_BLENDING block.

Also, it looks like lines 726 and 727 can be combined into one line.

guoqing-noaa · 2023-07-04T22:54:52Z

 SKEBNORM="1"
 SKEB_VDOF="10"
 USE_ZMTNBLCK="FALSE"
+BLENDING_LENGTHSCALE="960" # (Lx) in kilometers


Suggest changing to "ENS_BLENDING_LENGTHSCAL"

… ENS_BLENDING_LENGTHSCALE.

delippi · 2023-07-12T13:52:26Z

I'm going to close this PR and open an new PR in the new rrfs-workflow repo (https://github.com/NOAA-EMC/rrfs-workflow).

…w BKTYPE=3 (warm start from blended ics).

delippi · 2023-08-11T20:57:26Z

I initially closed this to put into the other workflow, but I'm not sure where that stands so I reopened this one to get this in quicker. I also made some updates especially in run prep script for BKTYPE=3 which is warm starting from blended ics.

guoqing-noaa · 2023-10-26T18:12:00Z

+   ln_vrfy -sf $EXECDIR/raymond.cpython-38-x86_64-linux-gnu.so .
+   ln_vrfy -sf $EXECDIR/chgres_winds.cpython-38-x86_64-linux-gnu.so .
+   ln_vrfy -sf $EXECDIR/remap_scalar.cpython-38-x86_64-linux-gnu.so .
+   ln_vrfy -sf $EXECDIR/remap_dwinds.cpython-38-x86_64-linux-gnu.so .


We can remove the "cpython-38-x86_64-linux-gnu" text now

@guoqing-noaa Right. Also, EXECDIR needs to be something like LIB64DIR? I don't see the path to lib64 defined anywhere.

I had thought about installing those *.so files into bin/.

With this, it will be good that we change CMAKE_INSTALL_LIBDIR to CMAKE_INSTALL_BINDIR when building your blending code. CMAKE_INSTALL_LIBDIR currently holds intermediate results instead of final results. And I think these four *so files are final results.

If you agree, you may create a PR to rrfs_utl to make changes and we don't need to change EXECDIR here. Thanks!

@guoqing-noaa, I added the LIB64DIR variable.

guoqing-noaa · 2023-10-26T18:12:41Z

+   bndy=./gfs.bndy.nc
+
+   # Run convert coldstart files to fv3 restart (rotate winds and remap).
+   python da_chgres_cold2fv3.py $warm $cold $grid $akbk $akbkcold $orog


change "python" to "${BLENDINGPYTHON}"

@guoqing-noaa, done.

guoqing-noaa · 2023-10-26T18:12:52Z

+   trcr=./fv_tracer.res.tile1.nc
+
+   # Blend.
+   python da_blending_fv3.py $Lx $glb $reg $trcr


change "python" to "${BLENDINGPYTHON}"

@guoqing-noaa, done.

…=false

guoqing-noaa

LGTM

hu5970 · 2023-11-02T20:54:08Z

@@ -0,0 +1,229 @@
+MACHINE="wcoss2"


This should be merged with config.sh_rrfs_a_enks_c3_retro

@hu5970 , I removed both of my blending specific configs and merged what was needed into config.sh_rrfs_a_enks_c3_retro.

hu5970 · 2023-11-02T20:54:54Z

@@ -0,0 +1,142 @@
+MACHINE="wcoss2"


This should be merged with config.sh_rrfs_a_c3_retro

hu5970 · 2023-11-02T20:59:04Z

    </dependency>
  </task>
 {%- endif %}
+{%- if do_ens_blending and not do_ensinit and do_spinup %}


This should be combined with the recenter task in line 1282 - 1316

@hu5970 , done. Thanks for this suggestion.

…moved blending specific configs.

Co-authored-by: donald lippi <donald.e.lippi@dlogin02.dogwood.wcoss2.ncep.noaa.gov>

Adding large-scale blending step to regional_workflow.

1f56f64

delippi requested review from CurtisAlexander-NOAA, JacobCarley-NOAA, JeffBeck-NOAA, JiliDong-NOAA, MatthewPyle-NOAA, ShunLiu-NOAA, TrevorAlcott-NOAA, christinaholtNOAA, chunhuazhou, daviddowellNOAA, guoqing-noaa, hu5970 and terraladwig as code owners June 28, 2023 19:48

delippi mentioned this pull request Jun 28, 2023

add blending and associated pre-processing tools. NOAA-GSL/rrfs_utl#38

Merged

JacobCarley-NOAA reviewed Jun 28, 2023

View reviewed changes

This comment was marked as duplicate.

Sign in to view

guoqing-noaa reviewed Jul 4, 2023

View reviewed changes

Rename DO_BLENDING to DO_ENS_BLENDING. Rename BLENDING_LENGTHSCALE to…

9b5d0c2

… ENS_BLENDING_LENGTHSCALE.

delippi closed this Jul 12, 2023

delippi and others added 4 commits July 14, 2023 13:58

Merge branch 'NOAA-GSL:feature/RRFS_dev1' into feature/RRFS_dev1

ae00f9f

Merge branch 'NOAA-GSL:feature/RRFS_dev1' into feature/RRFS_dev1

aca993b

Made some logic fixes to make ics. Added new logic to run prep for ne…

a09c37f

…w BKTYPE=3 (warm start from blended ics).

add config.sh files for large-scale blending retros.

815f9ec

delippi reopened this Aug 11, 2023

delippi mentioned this pull request Oct 26, 2023

Add "BLENDINGPYTHON" environment variable to env/build*intel*env for ... NOAA-GSL/ufs-srweather-app#258

Merged

Merge branch 'NOAA-GSL:feature/RRFS_dev1' into feature/RRFS_dev1

6b173ff

guoqing-noaa reviewed Oct 26, 2023

View reviewed changes

donald lippi added 4 commits October 30, 2023 16:44

Refactor scripts to work as blending or cold2warm conversion.

93192f7

Add LIB64DIR path for locating python *.so blending modules

07d0ac1

Add logic for do_ens_blending and to keep recentering when do_ensinit…

bff8998

…=false

Update blending config.sh files

82aa8d6

guoqing-noaa approved these changes Oct 30, 2023

View reviewed changes

hu5970 reviewed Nov 2, 2023

View reviewed changes

donald lippi added 2 commits November 3, 2023 14:13

combined recenter task in line 1282-1316.

2cccafa

merged blending parameters into config.sh_rrfs_a_enkf_c3_retro and re…

a2e9164

…moved blending specific configs.

hu5970 approved these changes Nov 6, 2023

View reviewed changes

guoqing-noaa merged commit fb2b56c into NOAA-GSL:feature/RRFS_dev1 Nov 6, 2023

delippi deleted the feature/RRFS_dev1 branch November 7, 2023 15:15

delippi restored the feature/RRFS_dev1 branch November 7, 2023 15:16

delippi pushed a commit to delippi/regional_workflow that referenced this pull request Nov 7, 2023

Add two settings for blending that were missed in NOAA-GSL#540.

ba9ef50

delippi mentioned this pull request Nov 7, 2023

Add two settings for blending that were missed in #540. #590

Merged

hu5970 pushed a commit that referenced this pull request Nov 9, 2023

Add two settings for blending that were missed in #540. (#590)

e4dd084

Co-authored-by: donald lippi <donald.e.lippi@dlogin02.dogwood.wcoss2.ncep.noaa.gov>

Conversation

delippi commented Jun 28, 2023

DESCRIPTION OF CHANGES:

TESTS CONDUCTED:

DEPENDENCIES:

DOCUMENTATION:

Uh oh!

JacobCarley-NOAA Jun 28, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

This comment was marked as duplicate.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Uh oh!

delippi commented Jul 12, 2023

Uh oh!

delippi commented Aug 11, 2023

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

guoqing-noaa left a comment

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

JacobCarley-NOAA Jun 28, 2023 •

edited

Loading