Skip to content

Add the RRFS_NA_3km pre-defined domain to the SRW App #480

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
JeffBeck-NOAA wants to merge 13 commits into from
Closed

Add the RRFS_NA_3km pre-defined domain to the SRW App #480

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
3f97757
Add EMC's RRFS_NA_3km domain
JeffBeck-NOAA Apr 22, 2021
2b1299f
Typo in case statement
JeffBeck-NOAA Apr 22, 2021
44fc906
Change CPUs per task to 2 instead of 4 to keep the number of requeste…
JeffBeck-NOAA Apr 27, 2021
02012e7
Reduce layout values based on EMC runs on Hera.
JeffBeck-NOAA Apr 27, 2021
d0479a3
Remove extra lines.
JeffBeck-NOAA Apr 27, 2021
49d6e73
Update k/n_split values based on EMC tests (substantially improves ru…
JeffBeck-NOAA Apr 27, 2021
1b64665
Update WRTCMP settings
JeffBeck-NOAA Apr 27, 2021
fee99df
Add a WE2E test for the RRFS_NA_3km domain
JeffBeck-NOAA Apr 27, 2021
4e9c2ff
Remove extra line.
JeffBeck-NOAA Apr 27, 2021
db50160
Update comments in config.sh file
JeffBeck-NOAA Apr 27, 2021
6de94f5
Update to comments.
JeffBeck-NOAA Apr 27, 2021
f996d2e
Remove blank lines.
JeffBeck-NOAA May 5, 2021
f6a973c
Merge branch 'feature/RRFS_NA_3km' of https://github.com/JeffBeck-NOA…
JeffBeck-NOAA May 5, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
49 changes: 49 additions & 0 deletions tests/baseline_configs/config.grid_RRFS_NA_3km.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
RUN_ENVIR="community"
PREEXISTING_DIR_METHOD="rename"

PREDEF_GRID_NAME="RRFS_NA_3km"
QUILTING="TRUE"

CCPP_PHYS_SUITE="FV3_GSD_SAR"
Copy link
Copy Markdown
Collaborator

@mkavulich mkavulich May 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

One more comment: Should this use the RRFS_v1alpha suite instead?

Copy link
Copy Markdown
Collaborator Author

@JeffBeck-NOAA JeffBeck-NOAA May 5, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, it probably should be. I can change it in the PR, but with the problems we're having with RRFS_v1alpha at the moment, we can't test with that suite (at least not on Hera).


FCST_LEN_HRS="06"
LBC_SPEC_INTVL_HRS="6"

DATE_FIRST_CYCL="20190701"
DATE_LAST_CYCL="20190701"
CYCL_HRS=( "00" )

EXTRN_MDL_NAME_ICS="FV3GFS"
EXTRN_MDL_NAME_LBCS="FV3GFS"
USE_USER_STAGED_EXTRN_FILES="TRUE"

#########################################################################
# The following code/namelist/workflow setting changes are necessary to #
# run/optimize end-to-end experiments using the 3-km NA grid #
#########################################################################

# The model should be built in 32-bit mode (64-bit will result in much
# longer run times.

# Use k_split=5 and n_split=2, the previous namelist values (k_split=4
# and n_split=5) will result in significantly longer run times.

# For Hera:
# 1) With 64-bit build and k_split=4/n_split=5, each time step takes ~21 seconds
# 2) 1) with k_split=5/n_split=2, each time step takes ~15 seconds
# 3) 2) with 32-bit build, each time step takes ~6 seconds

NNODES_MAKE_ICS="12"
NNODES_MAKE_LBCS="12"
PPN_MAKE_ICS="4"
PPN_MAKE_LBCS="4"
WTIME_MAKE_LBCS="01:00:00"

PPN_RUN_FCST="24"

NNODES_RUN_POST="6"
PPN_RUN_POST="12"

OMP_STACKSIZE_RUN_FCST="2048m"

###############################################################################
1 change: 1 addition & 0 deletions tests/baselines_list.txt
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -48,6 +48,7 @@ grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_GFS_v15p2
grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_GSD_SAR
grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_HRRR
grid_RRFS_CONUS_3km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
grid_RRFS_NA_3km
grid_RRFS_NA_13km
grid_RRFS_SUBCONUS_3km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16beta
grid_RRFS_SUBCONUS_3km_ics_HRRR_lbcs_RAP_suite_GFS_v15p2
Expand Down
2 changes: 1 addition & 1 deletion ush/config_defaults.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1657,7 +1657,7 @@ KMP_AFFINITY_RUN_FCST="scatter"
OMP_NUM_THREADS_RUN_FCST="4"
OMP_STACKSIZE_RUN_FCST="1024m"

CPUS_PER_TASK_RUN_FCST="4"
CPUS_PER_TASK_RUN_FCST="2"
Copy link
Copy Markdown
Collaborator

@mkavulich mkavulich May 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@JeffBeck-NOAA I'm testing this PR on Cheyenne right now. Aside from that, my only comment is about this line: Can you provide a brief justification for this change to the defaults in your PR message? This (along with the other changes you already mention) has the potential to impact others significantly.

Copy link
Copy Markdown
Collaborator Author

@JeffBeck-NOAA JeffBeck-NOAA May 4, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @mkavulich. Good catch on this. Setting this field to "4" was doubling the requested nodes for the run_fcst task. For example, a 3-km CONUS run that normally requests 25 nodes (based on predefined layout_x/y values) was asking for 50, simply because CPUS_PER_TASK_RUN_FCST=4, which was unacceptable. When it is set to "2", the number of nodes remains unchanged, in line with the layout_x/y values. EMC is using CPUS_PER_TASK_RUN_FCST=2 for their runs. I can add this to the PR message.


KMP_AFFINITY_RUN_POST="scatter"
OMP_NUM_THREADS_RUN_POST="1"
Expand Down
42 changes: 42 additions & 0 deletions ush/set_predef_grid_params.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1166,6 +1166,48 @@ case ${PREDEF_GRID_NAME} in
WRTCMP_dlat=$( printf "%.9f" $( bc -l <<< "(${ESGgrid_DELY}/${radius_Earth})*${degs_per_radian}" ) )
fi
;;
#
#-----------------------------------------------------------------------
#
# Future operational RRFS domain with ~3km cell size.
#
#-----------------------------------------------------------------------
#
"RRFS_NA_3km")

GRID_GEN_METHOD="ESGgrid"

ESGgrid_LON_CTR=-107.5
ESGgrid_LAT_CTR=51.5

ESGgrid_DELX="3000.0"
ESGgrid_DELY="3000.0"

ESGgrid_NX=3640
ESGgrid_NY=2520

ESGgrid_WIDE_HALO_WIDTH=6

DT_ATMOS="${DT_ATMOS:-36}"

LAYOUT_X="${LAYOUT_X:-18}"
LAYOUT_Y="${LAYOUT_Y:-36}"
BLOCKSIZE="${BLOCKSIZE:-28}"

if [ "$QUILTING" = "TRUE" ]; then
WRTCMP_write_groups="1"
WRTCMP_write_tasks_per_group="144"
WRTCMP_output_grid="rotated_latlon"
WRTCMP_cen_lon="-112.0" #${ESGgrid_LON_CTR}"
WRTCMP_cen_lat="48.0" #${ESGgrid_LAT_CTR}"
WRTCMP_lon_lwr_left="-51.0"
WRTCMP_lat_lwr_left="-33.0"
WRTCMP_lon_upr_rght="51.0"
WRTCMP_lat_upr_rght="33.0"
WRTCMP_dlon="0.025" #$( printf "%.9f" $( bc -l <<< "(${ESGgrid_DELX}/${radius_Earth})*${degs_per_radian}" ) )
WRTCMP_dlat="0.025" #$( printf "%.9f" $( bc -l <<< "(${ESGgrid_DELY}/${radius_Earth})*${degs_per_radian}" ) )
fi
;;

esac
#
Expand Down
2 changes: 0 additions & 2 deletions ush/setup.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -2265,8 +2265,6 @@ fi
#-----------------------------------------------------------------------
#
NNODES_RUN_FCST=$(( (PE_MEMBER01 + PPN_RUN_FCST - 1)/PPN_RUN_FCST ))


#
#-----------------------------------------------------------------------
#
Expand Down
4 changes: 2 additions & 2 deletions ush/templates/input.nml.FV3
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -85,15 +85,15 @@
hord_vt = 6
hydrostatic = .false.
io_layout = 1,1
k_split = 4
k_split = 5
ke_bg = 0.0
kord_mt = 9
kord_tm = -9
kord_tr = 9
kord_wz = 9
make_nh = .true.
mountain = .false.
n_split = 5
n_split = 2
n_sponge = 24
n_zs_filter = 0
na_init = 1
Expand Down
1 change: 1 addition & 0 deletions ush/valid_param_vals.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -20,6 +20,7 @@ valid_vals_PREDEF_GRID_NAME=( \
"GSL_HAFSV0.A_3km" \
"GSD_HRRR_AK_50km" \
"RRFS_NA_13km" \
"RRFS_NA_3km" \
)
valid_vals_CCPP_PHYS_SUITE=( \
"FV3_CPT_v0" \
Expand Down