Skip to content

Reinstate product groups #3208

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
WalterKolczynski-NOAA merged 26 commits into from
Jan 15, 2025
Merged

Reinstate product groups #3208

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
94143aa
Reinstate product groups
WalterKolczynski-NOAA Jan 6, 2025
c4782e2
Merge branch 'develop' into feature/prod_groups
WalterKolczynski-NOAA Jan 7, 2025
e741267
Add wave gridded post data dependency
WalterKolczynski-NOAA Jan 7, 2025
f6c476d
Fix documentation typos in tasks.py
WalterKolczynski-NOAA Jan 7, 2025
7287f0f
Use fhr in jobid for product jobs
WalterKolczynski-NOAA Jan 7, 2025
b8c832d
Add missing whitespace
WalterKolczynski-NOAA Jan 8, 2025
f0444d5
Add comment about removing output hours for replay
WalterKolczynski-NOAA Jan 8, 2025
d66e3c0
Merge branch 'develop' into feature/prod_groups
aerorahul Jan 8, 2025
23b9850
Merge branch 'develop' into feature/prod_groups
WalterKolczynski-NOAA Jan 8, 2025
833cb44
Move new task tests to separate file
WalterKolczynski-NOAA Jan 8, 2025
bc07a7f
Switch from unittest to pytest for Tasks test
WalterKolczynski-NOAA Jan 8, 2025
d6b5a5c
Extend grouping to ensstat
WalterKolczynski-NOAA Jan 8, 2025
898607c
Reset permissions on workflow tests
WalterKolczynski-NOAA Jan 8, 2025
d6492ad
Move workflow tests to tests directory
WalterKolczynski-NOAA Jan 8, 2025
9372fd2
Update ensstat rocoto stub to loop over fhr
WalterKolczynski-NOAA Jan 8, 2025
c24412d
Replace tabs with spaces in tasks test
WalterKolczynski-NOAA Jan 8, 2025
9f05093
Fix ensstat exit code
WalterKolczynski-NOAA Jan 8, 2025
a6ed2fd
Switch type hint from pipe to Union for backwards compatability
WalterKolczynski-NOAA Jan 9, 2025
77b7312
Use bisect instead of index to split fhrs
WalterKolczynski-NOAA Jan 9, 2025
0dccf85
Remove empty arrays after breakpoint splitting
WalterKolczynski-NOAA Jan 9, 2025
db1226d
Break atmos gempak into groups
WalterKolczynski-NOAA Jan 10, 2025
7f978e6
Add comment about walltime for ensstat
WalterKolczynski-NOAA Jan 10, 2025
98d6050
Fix copy/paste error
WalterKolczynski-NOAA Jan 10, 2025
48c1acb
Merge branch 'develop' into feature/prod_groups
aerorahul Jan 10, 2025
bae8d2d
Fix gempak stub
WalterKolczynski-NOAA Jan 14, 2025
cae19d4
Merge branch 'develop' into feature/prod_groups
WalterKolczynski-NOAA Jan 14, 2025
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
21 changes: 14 additions & 7 deletions jobs/rocoto/atmos_ensstat.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,20 @@ status=$?
if (( status != 0 )); then exit "${status}"; fi

export job="atmos_ensstat"
export jobid="${job}.$$"

export FORECAST_HOUR=$(( 10#${FHR3} ))
# shellcheck disable=SC2153
IFS=', ' read -r -a fhr_list <<< "${FHR_LIST}"

###############################################################
# Execute the JJOB
###############################################################
"${HOMEgfs}/jobs/JGLOBAL_ATMOS_ENSSTAT"
export FORECAST_HOUR jobid
for FORECAST_HOUR in "${fhr_list[@]}"; do
fhr3=$(printf '%03d' "${FORECAST_HOUR}")
jobid="${job}_f${fhr3}.$$"
###############################################################
# Execute the JJOB
###############################################################
"${HOMEgfs}/jobs/JGLOBAL_ATMOS_ENSSTAT"
status=$?
[[ ${status} -ne 0 ]] && exit "${status}"
done

exit $?
exit 0
23 changes: 14 additions & 9 deletions jobs/rocoto/atmos_products.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,15 +13,20 @@ status=$?
if (( status != 0 )); then exit "${status}"; fi

export job="atmos_products"
export jobid="${job}.$$"

# Negatation needs to be before the base
fhr3_base="10#${FHR3}"
export FORECAST_HOUR=$(( ${fhr3_base/10#-/-10#} ))
# shellcheck disable=SC2153
IFS=', ' read -r -a fhr_list <<< "${FHR_LIST}"

###############################################################
# Execute the JJOB
###############################################################
"${HOMEgfs}/jobs/JGLOBAL_ATMOS_PRODUCTS"
export FORECAST_HOUR jobid
for FORECAST_HOUR in "${fhr_list[@]}"; do
fhr3=$(printf '%03d' "${FORECAST_HOUR}")
jobid="${job}_f${fhr3}.$$"
###############################################################
# Execute the JJOB
###############################################################
"${HOMEgfs}/jobs/JGLOBAL_ATMOS_PRODUCTS"
status=$?
Comment thread
WalterKolczynski-NOAA marked this conversation as resolved.
[[ ${status} -ne 0 ]] && exit "${status}"
done

exit $?
exit 0
19 changes: 14 additions & 5 deletions jobs/rocoto/gempak.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,20 @@ status=$?
if (( status != 0 )); then exit "${status}"; fi

export job="gempak"
export jobid="${job}.$$"

# shellcheck disable=SC2153
IFS=', ' read -r -a fhr_list <<< "${FHR_LIST}"

# Execute the JJOB
"${HOMEgfs}/jobs/J${RUN^^}_ATMOS_GEMPAK"
export FHR3 jobid
for fhr in "${fhr_list[@]}"; do
FHR3=$(printf '%03d' "${fhr}")
jobid="${job}_f${FHR3}.$$"
###############################################################
# Execute the JJOB
###############################################################
"${HOMEgfs}/jobs/J${RUN^^}_ATMOS_GEMPAK"
status=$?
[[ ${status} -ne 0 ]] && exit "${status}"
done

status=$?
exit "${status}"
exit 0
21 changes: 14 additions & 7 deletions jobs/rocoto/oceanice_products.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,13 +13,20 @@ status=$?
if (( status != 0 )); then exit "${status}"; fi

export job="oceanice_products"
export jobid="${job}.$$"

export FORECAST_HOUR=$(( 10#${FHR3} ))
# shellcheck disable=SC2153
IFS=', ' read -r -a fhr_list <<< "${FHR_LIST}"

###############################################################
# Execute the JJOB
###############################################################
"${HOMEgfs}/jobs/JGLOBAL_OCEANICE_PRODUCTS"
export FORECAST_HOUR jobid
for FORECAST_HOUR in "${fhr_list[@]}"; do
fhr3=$(printf '%03d' "${FORECAST_HOUR}")
jobid="${job}_${COMPONENT}_f${fhr3}.$$"
###############################################################
# Execute the JJOB
###############################################################
"${HOMEgfs}/jobs/JGLOBAL_OCEANICE_PRODUCTS"
Comment thread
WalterKolczynski-NOAA marked this conversation as resolved.
status=$?
[[ ${status} -ne 0 ]] && exit "${status}"
done

exit $?
exit 0
21 changes: 14 additions & 7 deletions jobs/rocoto/wavepostsbs.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,17 +5,24 @@ source "${HOMEgfs}/ush/preamble.sh"
###############################################################
# Source FV3GFS workflow modules
#. ${HOMEgfs}/ush/load_fv3gfs_modules.sh
. ${HOMEgfs}/ush/load_ufswm_modules.sh
source "${HOMEgfs}/ush/load_ufswm_modules.sh"
status=$?
[[ ${status} -ne 0 ]] && exit ${status}
[[ ${status} -ne 0 ]] && exit "${status}"

export job="wavepostsbs"
export jobid="${job}.$$"

###############################################################
# Execute the JJOB
${HOMEgfs}/jobs/JGLOBAL_WAVE_POST_SBS
status=$?
[[ ${status} -ne 0 ]] && exit ${status}
# shellcheck disable=SC2153
IFS=', ' read -r -a fhr_list <<< "${FHR_LIST}"

export FHR3 jobid
for FORECAST_HOUR in "${fhr_list[@]}"; do
FHR3=$(printf '%03d' "${FORECAST_HOUR}")
jobid="${job}_f${FHR3}.$$"
# Execute the JJOB
"${HOMEgfs}/jobs/JGLOBAL_WAVE_POST_SBS"
Comment thread
WalterKolczynski-NOAA marked this conversation as resolved.
status=$?
[[ ${status} -ne 0 ]] && exit "${status}"
done

exit 0
3 changes: 3 additions & 0 deletions parm/config/gefs/config.atmos_ensstat
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,6 +5,9 @@

echo "BEGIN: config.atmos_ensstat"

# Maximum number of rocoto tasks
export MAX_TASKS=25

# Get task specific resources
. "${EXPDIR}/config.resources" atmos_ensstat

Expand Down
4 changes: 2 additions & 2 deletions parm/config/gefs/config.atmos_products
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ echo "BEGIN: config.atmos_products"
# Get task specific resources
. "${EXPDIR}/config.resources" atmos_products

# No. of forecast hours to process in a single job
export NFHRS_PER_GROUP=3
# Maximum number of rocoto tasks per member
export MAX_TASKS=25

# Scripts used by this job
export INTERP_ATMOS_MASTERSH="${USHgfs}/interp_atmos_master.sh"
Expand Down
4 changes: 2 additions & 2 deletions parm/config/gefs/config.oceanice_products
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,7 +9,7 @@ source "${EXPDIR}/config.resources" oceanice_products

export OCEANICEPRODUCTS_CONFIG="${PARMgfs}/post/oceanice_products_gefs.yaml"

# No. of forecast hours to process in a single job
export NFHRS_PER_GROUP=3
# Maximum number of rocoto tasks per member
export MAX_TASKS=25

echo "END: config.oceanice_products"
10 changes: 7 additions & 3 deletions parm/config/gefs/config.resources
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -234,6 +234,7 @@ case ${step} in
;;

"atmos_products")
# Walltime is per forecast hour; will be multipled by group size
export walltime="00:15:00"
export ntasks=24
export threads_per_task=1
Expand All @@ -242,14 +243,16 @@ case ${step} in
;;

"atmos_ensstat")
Comment thread
WalterKolczynski-NOAA marked this conversation as resolved.
export walltime="00:30:00"
# Walltime is per forecast hour; will be multipled by group size
export walltime="00:15:00"
export ntasks=6
export threads_per_task=1
export tasks_per_node="${ntasks}"
export is_exclusive=True
;;

"oceanice_products")
# Walltime is per forecast hour; will be multipled by group size
export walltime="00:15:00"
export ntasks=1
export tasks_per_node=1
Expand All @@ -258,7 +261,8 @@ case ${step} in
;;

"wavepostsbs")
export walltime="03:00:00"
# Walltime is per forecast hour; will be multipled by group size
export walltime="00:15:00"
export ntasks=1
export threads_per_task=1
export tasks_per_node=$(( max_tasks_per_node / threads_per_task ))
Expand Down Expand Up @@ -328,7 +332,7 @@ case ${step} in
;;

"cleanup")
export walltime="00:15:00"
export walltime="00:30:00"
export ntasks=1
export tasks_per_node=1
export threads_per_task=1
Expand Down
3 changes: 3 additions & 0 deletions parm/config/gefs/config.wavepostsbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ echo "BEGIN: config.wavepostsbs"
# Get task specific resources
source "${EXPDIR}/config.resources" wavepostsbs

# Maximum number of rocoto tasks per member
export MAX_TASKS=25

# Subgrid info for grib2 encoding
export WAV_SUBGRBSRC=""
export WAV_SUBGRB=""
Expand Down
4 changes: 2 additions & 2 deletions parm/config/gfs/config.atmos_products
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,8 +8,8 @@ echo "BEGIN: config.atmos_products"
# Get task specific resources
. "${EXPDIR}/config.resources" atmos_products

# No. of forecast hours to process in a single job
export NFHRS_PER_GROUP=3
## Maximum number of rocoto tasks per member
export MAX_TASKS=25

# Scripts used by this job
export INTERP_ATMOS_MASTERSH="${USHgfs}/interp_atmos_master.sh"
Expand Down
5 changes: 4 additions & 1 deletion parm/config/gfs/config.gempak
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,7 +5,10 @@

echo "BEGIN: config.gempak"

# Maximum number of rocoto tasks per member
export MAX_TASKS=25

# Get task specific resources
. $EXPDIR/config.resources gempak
source "${EXPDIR}/config.resources" gempak

echo "END: config.gempak"
3 changes: 3 additions & 0 deletions parm/config/gfs/config.oceanice_products
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,6 +7,9 @@ echo "BEGIN: config.oceanice_products"
# Get task specific resources
source "${EXPDIR}/config.resources" oceanice_products

# Maximum number of rocoto tasks per member
export MAX_TASKS=25

export OCEANICEPRODUCTS_CONFIG="${PARMgfs}/post/oceanice_products.yaml"

# No. of forecast hours to process in a single job
Expand Down
8 changes: 6 additions & 2 deletions parm/config/gfs/config.resources
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -191,8 +191,9 @@ case ${step} in
;;

"wavepostsbs")
walltime_gdas="00:20:00"
walltime_gfs="03:00:00"
# Walltime is per forecast hour; will be multipled by group size
walltime_gdas="00:15:00"
walltime_gfs="00:15:00"
ntasks=8
threads_per_task=1
tasks_per_node=$(( max_tasks_per_node / threads_per_task ))
Expand Down Expand Up @@ -911,6 +912,7 @@ case ${step} in
;;

"oceanice_products")
# Walltime is per forecast hour; will be multipled by group size
walltime="00:15:00"
ntasks=1
tasks_per_node=1
Expand Down Expand Up @@ -949,6 +951,7 @@ case ${step} in
;;

"atmos_products")
# Walltime is per forecast hour; will be multipled by group size
walltime="00:15:00"
ntasks=24
threads_per_task=1
Expand Down Expand Up @@ -1278,6 +1281,7 @@ case ${step} in
;;

"gempak")
# Walltime is per forecast hour; will be multipled by group size
walltime="00:30:00"
ntasks_gdas=2
ntasks_gfs=28
Expand Down
3 changes: 3 additions & 0 deletions parm/config/gfs/config.wavepostsbs
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,6 +8,9 @@ echo "BEGIN: config.wavepostsbs"
# Get task specific resources
source "${EXPDIR}/config.resources" wavepostsbs

# Maximum number of rocoto tasks per member
export MAX_TASKS=25

# Subgrid info for grib2 encoding
export WAV_SUBGRBSRC=""
export WAV_SUBGRB=""
Expand Down
Loading