Skip to content

Abstract common j-job tasks #1230

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
WalterKolczynski-NOAA merged 11 commits into from
Jan 14, 2023
Merged

Abstract common j-job tasks #1230

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
11 commits
Select commit Hold shift + click to select a range
e46dbf5
Abstract common j-job tasks
WalterKolczynski-NOAA Jan 6, 2023
dcebeee
Clean $DATA and have all steps of ocean anl use same $DATA
WalterKolczynski-NOAA Jan 9, 2023
0d45879
Only add CPU ranks to gempak CFP file when needed
WalterKolczynski-NOAA Jan 10, 2023
0afc2c7
Fix linter complaint in special gempak
WalterKolczynski-NOAA Jan 10, 2023
16c1e75
Add CFP_MP changes to env for gempak
WalterKolczynski-NOAA Jan 10, 2023
a33654e
Specify job for environment script
WalterKolczynski-NOAA Jan 12, 2023
4e55e15
Remove ability to override $DATA for ocean analysis
WalterKolczynski-NOAA Jan 12, 2023
cac3b2f
Fix reading of config array in jjob header
WalterKolczynski-NOAA Jan 12, 2023
90a060b
Fix reading in array of config files
WalterKolczynski-NOAA Jan 12, 2023
29ffd9a
Fix GLDAS cycle checking
WalterKolczynski-NOAA Jan 13, 2023
ede2b0d
Add missing exit when job is not provided to header
WalterKolczynski-NOAA Jan 13, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 2 additions & 0 deletions env/HERA.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -290,6 +290,8 @@ elif [[ "${step}" = "awips" ]]; then

elif [[ "${step}" = "gempak" ]]; then

export CFP_MP="YES"

nth_max=$((npe_node_max / npe_node_gempak))

export NTHREADS_GEMPAK=${nth_gempak:-1}
Expand Down
2 changes: 2 additions & 0 deletions env/ORION.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -288,6 +288,8 @@ elif [[ "${step}" = "awips" ]]; then

elif [[ "${step}" = "gempak" ]]; then

export CFP_MP="YES"

nth_max=$((npe_node_max / npe_node_gempak))

export NTHREADS_GEMPAK=${nth_gempak:-1}
Expand Down
44 changes: 1 addition & 43 deletions jobs/JGDAS_ATMOS_ANALYSIS_DIAG
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,49 +1,7 @@
#! /usr/bin/env bash

source "${HOMEgfs}/ush/preamble.sh"

##############################################
# make temp directory
##############################################
export DATA=${DATA:-${DATAROOT}/${jobid}}
mkdir -p ${DATA}
cd ${DATA}


##############################################
# Run setpdy and initialize PDY variables
##############################################
export cycle="t${cyc}z"
setpdy.sh
. ./PDY


##############################################
# Determine Job Output Name on System
##############################################
export pid=${pid:-$$}
export pgmout="OUTPUT.${pid}"
export pgmerr=errfile


#############################
# Source relevant config files
#############################
export EXPDIR=${EXPDIR:-${HOMEgfs}/parm/config}
configs="base anal analdiag"
for config in ${configs}; do
. ${EXPDIR}/config.${config}
status=$?
[[ ${status} -ne 0 ]] && exit ${status}
done


##########################################
# Source machine runtime environment
##########################################
. ${HOMEgfs}/env/${machine}.env anal
Comment thread
WalterKolczynski-NOAA marked this conversation as resolved.
status=$?
[[ ${status} -ne 0 ]] && exit ${status}
source "${HOMEgfs}/ush/jjob_header.sh" -e "anal" -c "base anal analdiag"


##############################################
Expand Down
44 changes: 1 addition & 43 deletions jobs/JGDAS_ATMOS_CHGRES_FORENKF
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,49 +1,7 @@
#! /usr/bin/env bash

source "${HOMEgfs}/ush/preamble.sh"

##############################################
# make temp directory
##############################################
export DATA=${DATA:-${DATAROOT}/${jobid}}
mkdir -p ${DATA}
cd ${DATA}


##############################################
# Run setpdy and initialize PDY variables
##############################################
export cycle="t${cyc}z"
setpdy.sh
. ./PDY


##############################################
# Determine Job Output Name on System
##############################################
export pid=${pid:-$$}
export pgmout="OUTPUT.${pid}"
export pgmerr=errfile


#############################
# Source relevant config files
#############################
export EXPDIR=${EXPDIR:-${HOMEgfs}/parm/config}
configs="base anal echgres"
for config in ${configs}; do
. ${EXPDIR}/config.${config}
status=$?
[[ ${status} -ne 0 ]] && exit ${status}
done


##########################################
# Source machine runtime environment
##########################################
. ${HOMEgfs}/env/${machine}.env anal
status=$?
[[ ${status} -ne 0 ]] && exit ${status}
source "${HOMEgfs}/ush/jjob_header.sh" -e "anal" -c "base anal echgres"


##############################################
Expand Down
41 changes: 3 additions & 38 deletions jobs/JGDAS_ATMOS_GEMPAK
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,45 +1,12 @@
#! /usr/bin/env bash

source "${HOMEgfs}/ush/preamble.sh"
source "${HOMEgfs}/ush/jjob_header.sh" -e "gempak" -c "base gempak"

##########################################################
# make temp directory
##########################################################
export DATA=${DATA:-${DATAROOT}/${jobid}}
mkdir -p ${DATA}
cd ${DATA}

###########################################
# Run setpdy and initialize PDY variables
###########################################
export cycle="t${cyc}z"
setpdy.sh
. ./PDY

export pid=${pid:-$$}
export pgmout="OUTPUT.${pid}"
export pgmerr=errfile

#############################
# Source relevant config files
#############################
export EXPDIR=${EXPDIR:-${HOMEgfs}/parm/config}
configs="base gempak"
for config in ${configs}; do
. ${EXPDIR}/config.${config}
status=$?
[[ ${status} -ne 0 ]] && exit ${status}
done

###############################################################
. ${BASE_ENV}/${machine}.env gempak
status=$?
[[ ${status} -ne 0 ]] && exit ${status}
# TODO (#1219) This j-job is not part of the rocoto suite

################################
# Set up the HOME directory
################################
export HOMEgfs=${HOMEgfs:-${PACKAGEROOT}/gfs.${gfs_ver}} # TODO: remove these as they are already defined in config.base
export EXECgfs=${EXECgfs:-${HOMEgfs}/exec}
export PARMgfs=${PARMgfs:-${HOMEgfs}/parm}
export PARMwmo=${PARMwmo:-${HOMEgfs}/parm/wmo}
Expand Down Expand Up @@ -67,8 +34,6 @@ export DBNROOT=${DBNROOT:-${UTILROOT}/fakedbn}
###################################
# Specify NET and RUN Name and model
####################################
export NET=${NET:-gfs}
export RUN=${RUN:-gdas}
export model=${model:-gdas}
export COMPONENT="atmos"

Expand All @@ -83,7 +48,7 @@ if [ ${SENDCOM} = YES ] ; then
fi


# TODO: These actions belong in an ex-script not a j-job
# TODO: These actions belong in an ex-script not a j-job (#1219)
if [ -f ${DATA}/poescript ]; then
rm ${DATA}/poescript
fi
Expand Down
23 changes: 3 additions & 20 deletions jobs/JGDAS_ATMOS_GEMPAK_META_NCDC
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,28 +1,13 @@
#! /usr/bin/env bash

source "${HOMEgfs}/ush/preamble.sh"

############################################
# GDAS GEMPAK META NCDC PRODUCT GENERATION
############################################

##########################################################
# make temp directory
##########################################################
export DATA=${DATA:-${DATAROOT}/${jobid}}
mkdir -p ${DATA}
cd ${DATA}
# TODO (#1222) This j-job is not part of the rocoto

###########################################
# Run setpdy and initialize PDY variables
###########################################
export cycle="t${cyc}z"
setpdy.sh
. ./PDY

export pid=${pid:-$$}
export pgmout="OUTPUT.${pid}"
export pgmerr=errfile
source "${HOMEgfs}/ush/preamble.sh"
source "${HOMEgfs}/ush/jjob_header.sh" -e "gempak_meta" -c "base gempak"

################################
# Set up the HOME directory
Expand All @@ -46,8 +31,6 @@ cp ${FIXgempak}/datatype.tbl datatype.tbl
###################################
# Specify NET and RUN Name and model
####################################
export NET=${NET:-gfs}
export RUN=${RUN:-gdas}
export COMPONENT="atmos"
export MODEL=GDAS
export GRID_NAME=gdas
Expand Down
48 changes: 3 additions & 45 deletions jobs/JGDAS_ATMOS_GLDAS
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,45 +1,11 @@
#! /usr/bin/env bash

source "${HOMEgfs:?}/ush/preamble.sh"

##############################################
# Obtain unique process id (pid) and make temp directory
##############################################
export DATA=${DATA:-${DATAROOT}/${jobid}}
mkdir -p "${DATA}"
cd "${DATA}" || exit 1


##############################################
# Run setpdy and initialize PDY variables
##############################################
export cycle="t${cyc}z"
setpdy.sh
. ./PDY


##############################################
# Determine Job Output Name on System
##############################################
export pid=${pid:-$$}
export pgmout="OUTPUT.${pid}"
export pgmerr=errfile


#############################
# Source relevant config files
#############################
export EXPDIR=${EXPDIR:-${HOMEgfs}/parm/config}
configs="base gldas"
for config in ${configs}; do
. ${EXPDIR}/config.${config}
status=$?
[[ ${status} -ne 0 ]] && exit "${status}"
done

source "${HOMEgfs}/ush/jjob_header.sh" -e "gldas" -c "base gldas"

if [[ "${cyc:?}" -ne "${gldas_cyc:?}" ]]; then
echo "GLDAS only runs for ${gldas_cyc} cycle; Skip GLDAS step for cycle ${cyc}"
rm -Rf "${DATA}"
exit 0
fi

Expand All @@ -48,18 +14,10 @@ xtime=$((gldas_spinup_hours+12))
if [[ "${CDATE}" -le "$(${NDATE:?} +"${xtime}" "${SDATE:?}")" ]]; then
echo "GLDAS needs fluxes as forcing from cycles in previous ${xtime} hours"
echo "starting from ${SDATE}. This gldas cycle is skipped"
rm -Rf "${DATA}"
exit 0
fi

##########################################
# Source machine runtime environment
##########################################
# shellcheck disable=SC1090-SC1091
. "${HOMEgfs}/env/${machine:?}.env" gldas
# shellcheck disable=
status=$?
[[ ${status} -ne 0 ]] && exit "${status}"


##############################################
# Set variables used in the exglobal script
Expand Down
26 changes: 1 addition & 25 deletions jobs/JGDAS_ATMOS_VERFOZN
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,36 +4,12 @@
# Set up environment for GDAS Ozone Monitor job
#############################################################
source "${HOMEgfs}/ush/preamble.sh"

###########################################################
# make temp directories
###########################################################
export DATA=${DATA:-${DATAROOT}/${jobid}}
mkdir -p ${DATA}
cd ${DATA}


##############################################
# Run setpdy and initialize PDY variables
##############################################
export cycle="t${cyc}z"
setpdy.sh
. ./PDY


####################################
# Determine Job Output Name on System
####################################
export pid=$$
export pgmout="OUTPUT.${pid}"
export pgmerr=errfile
source "${HOMEgfs}/ush/jjob_header.sh" -e "vrfy" -c "base vrfy"


###############################
# Specify NET, RUN, and COMPONENT name
##############################
export NET=${NET:-gfs}
export RUN=${RUN:-gdas}
export COMPONENT="atmos"

export OZNMON_SUFFIX=${OZNMON_SUFFIX:-${NET}}
Expand Down
30 changes: 1 addition & 29 deletions jobs/JGDAS_ATMOS_VERFRAD
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,36 +4,8 @@
# Set up environment for GDAS Radiance Monitor job
#############################################################
source "${HOMEgfs}/ush/preamble.sh"
source "${HOMEgfs}/ush/jjob_header.sh" -e "vrfy" -c "base vrfy"

###########################################################
# make temp directories
###########################################################
export DATA=${DATA:-${DATAROOT}/${jobid}}
mkdir -p ${DATA}
cd ${DATA}


##############################################
# Run setpdy and initialize PDY variables
##############################################
export cycle="t${cyc}z"
setpdy.sh
. ./PDY


####################################
# Determine Job Output Name on System
####################################
export pid=$$
export pgmout="OUTPUT.${pid}"
export pgmerr=errfile


###############################
# Specify NET, RUN, and COMPONENT name
##############################
export NET=${NET:-gfs}
export RUN=${RUN:-gdas}
export COMPONENT="atmos"

export RAD_DATA_IN=${DATA}
Expand Down
Loading