Skip to content

RRFS_dev1: update clean stratigy #123

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
hu5970 merged 1 commit into from
May 26, 2021
Merged

RRFS_dev1: update clean stratigy #123

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
1 commit
Select commit
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 37 additions & 13 deletions scripts/exregional_clean.ksh
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,58 +1,82 @@
#!/bin/ksh --login

currentime=`date`

#
#-----------------------------------------------------------------------
# Source the variable definitions file.
#-----------------------------------------------------------------------
#
. ${GLOBAL_VAR_DEFNS_FP}
#
#-----------------------------------------------------------------------
# Save current shell options (in a global array). Then set new options
# for this script/function.
#-----------------------------------------------------------------------
#
{ save_shell_opts; set -u -x; } > /dev/null 2>&1
#
#
#-----------------------------------------------------------------------
# set up currentime from CDATE
#-----------------------------------------------------------------------
#
currentime=$(echo "${CDATE}" | sed 's/\([[:digit:]]\{2\}\)$/ \1/')

#-----------------------------------------------------------------------
# Delete ptmp directories
deletetime=`date +%Y%m%d -d "${currentime} 72 hours ago"`
#-----------------------------------------------------------------------
deletetime=$(date +%Y%m%d -d "${currentime} ${CLEAN_OLDPROD_HRS} hours ago")
echo "Deleting ptmp directories before ${deletetime}..."
cd ${COMOUT_BASEDIR}
set -A XX `ls -d ${RUN}.20* | sort -r`
set -A XX $(ls -d ${RUN}.20* | sort -r)
for dir in ${XX[*]};do
onetime=`echo $dir | cut -d'.' -f2`
onetime=$(echo $dir | cut -d'.' -f2)
if [[ ${onetime} -le ${deletetime} ]]; then
rm -rf ${COMOUT_BASEDIR}/${RUN}.${onetime}
echo "Deleted ${COMOUT_BASEDIR}/${RUN}.${onetime}"
fi
done

#-----------------------------------------------------------------------
# Delete stmp directories
deletetime=`date +%Y%m%d%H -d "${currentime} 72 hours ago"`
#-----------------------------------------------------------------------
deletetime=$(date +%Y%m%d%H -d "${currentime} ${CLEAN_OLDRUN_HRS} hours ago")
echo "Deleting stmp directories before ${deletetime}..."
cd ${CYCLE_BASEDIR}
set -A XX `ls -d 20* | sort -r`
set -A XX $(ls -d 20* | sort -r)
for onetime in ${XX[*]};do
if [[ ${onetime} -le ${deletetime} ]]; then
Copy link
Copy Markdown
Collaborator

@guoqing-noaa guoqing-noaa May 26, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

To be more robust, suggest change line 33, 47 and similar if statements to
if [[ ${onetime} =~ '^[0-9]+$' ]] && [[ ${onetime} -le ${deletetime} ]]; then
At the RAP/HRRR era, there was one time when someone changed, for example, 20190508 to 20190508.save for debug purpose and then it crashed clean scripts for the next few days (maybe more than 10 days) , filled up the run disk very quickly.

rm -rf ${CYCLE_BASEDIR}/${onetime}
echo "Deleted ${CYCLE_BASEDIR}/${onetime}"
fi
done

#-----------------------------------------------------------------------
# Delete netCDF files
deletetime=`date +%Y%m%d%H -d "${currentime} 24 hours ago"`
#-----------------------------------------------------------------------
deletetime=$(date +%Y%m%d%H -d "${currentime} ${CLEAN_OLDFCST_HRS} hours ago")
echo "Deleting netCDF files before ${deletetime}..."
cd ${CYCLE_BASEDIR}
set -A XX `ls -d 20* | sort -r`
set -A XX $(ls -d 20* | sort -r)
for onetime in ${XX[*]};do
if [[ ${onetime} -le ${deletetime} ]]; then
rm -f ${CYCLE_BASEDIR}/${onetime}/fcst_fv3lam/phy*nc
rm -f ${CYCLE_BASEDIR}/${onetime}/fcst_fv3lam/dyn*nc
rm -rf ${CYCLE_BASEDIR}/${onetime}/fcst_fv3lam/RESTART
rm -rf ${CYCLE_BASEDIR}/${onetime}/fcst_fv3lam/INPUT
echo "Deleted netCDF files in ${CYCLE_BASEDIR}/${onetime}/fcst_fv3lam"
fi
done

#-----------------------------------------------------------------------
# Delete old log files
deletetime=`date +%Y%m%d%H -d "${currentime} 48 hours ago"`
#-----------------------------------------------------------------------
deletetime=$(date +%Y%m%d%H -d "${currentime} ${CLEAN_OLDLOG_HRS} hours ago")
echo "Deleting log files before ${deletetime}..."

# Remove template date from last two levels
logs=`echo ${LOGDIR} | rev | cut -f 3- -d / | rev`
logs=$(echo ${LOGDIR} | rev | cut -f 3- -d / | rev)
cd ${logs}
pwd
set -A XX `ls -d ${RUN}.20*/* | sort -r`
set -A XX $(ls -d ${RUN}.20*/* | sort -r)
for onetime in ${XX[*]}; do
# Remove slash and RUN from directory to get time
filetime=${onetime/\//}
Expand Down
11 changes: 10 additions & 1 deletion ush/config.sh.RRFS_AK_dev1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,16 @@ COMINgfs=""
STMP="/lfs4/BMC/nrtrr/NCO_dirs/stmp" # Path to directory STMP that mostly contains input files.
PTMP="/lfs4/BMC/nrtrr/NCO_dirs/ptmp" # Path to directory STMP that mostly contains input files.


if [[ $DO_RETRO == "true" ]] ; then
CLEAN_OLDPROD_HRS="240"
CLEAN_OLDLOG_HRS="240"
CLEAN_OLDRUN_HRS="6"
CLEAN_OLDFCST_HRS="6"
if [[ $LBCS_ICS_ONLY == "true" ]]; then
CLEAN_OLDRUN_HRS="7777"
CLEAN_OLDFCST_HRS="7777"
fi
fi
#
# In NCO mode, the user must manually (e.g. after doing the build step)
# create the symlink "${FIXrrfs}/fix_sar" that points to EMC's FIXLAM
Expand Down
11 changes: 11 additions & 0 deletions ush/config.sh.RRFS_NA_13km
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -128,6 +128,17 @@ COMINgfs=""
STMP="/lfs4/BMC/nrtrr/NCO_dirs/stmp" # Path to directory STMP that mostly contains input files.
PTMP="/lfs4/BMC/nrtrr/NCO_dirs/ptmp" # Path to directory STMP that mostly contains input files.

if [[ $DO_RETRO == "true" ]] ; then
CLEAN_OLDPROD_HRS="240"
CLEAN_OLDLOG_HRS="240"
CLEAN_OLDRUN_HRS="6"
CLEAN_OLDFCST_HRS="6"
if [[ $LBCS_ICS_ONLY == "true" ]]; then
CLEAN_OLDRUN_HRS="7777"
CLEAN_OLDFCST_HRS="7777"
fi
fi

#
# In NCO mode, the user must manually (e.g. after doing the build step)
# create the symlink "${FIXrrfs}/fix_sar" that points to EMC's FIXLAM
Expand Down
11 changes: 11 additions & 0 deletions ush/config.sh.RRFS_NA_3km
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -137,6 +137,17 @@ COMINgfs=""
STMP="/lfs4/BMC/nrtrr/NCO_dirs/stmp" # Path to directory STMP that mostly contains input files.
PTMP="/lfs4/BMC/nrtrr/NCO_dirs/ptmp" # Path to directory STMP that mostly contains input files.

if [[ $DO_RETRO == "true" ]] ; then
CLEAN_OLDPROD_HRS="240"
CLEAN_OLDLOG_HRS="240"
CLEAN_OLDRUN_HRS="6"
CLEAN_OLDFCST_HRS="6"
if [[ $LBCS_ICS_ONLY == "true" ]]; then
CLEAN_OLDRUN_HRS="7777"
CLEAN_OLDFCST_HRS="7777"
fi
fi

#
# In NCO mode, the user must manually (e.g. after doing the build step)
# create the symlink "${FIXrrfs}/fix_sar" that points to EMC's FIXLAM
Expand Down
11 changes: 11 additions & 0 deletions ush/config.sh.RRFS_dev1
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -150,6 +150,17 @@ COMINgfs=""
STMP="/lfs4/BMC/nrtrr/NCO_dirs/stmp" # Path to directory STMP that mostly contains input files.
PTMP="/lfs4/BMC/nrtrr/NCO_dirs/ptmp" # Path to directory STMP that mostly contains input files.

if [[ $DO_RETRO == "true" ]] ; then
CLEAN_OLDPROD_HRS="240"
CLEAN_OLDLOG_HRS="240"
CLEAN_OLDRUN_HRS="6"
CLEAN_OLDFCST_HRS="6"
if [[ $LBCS_ICS_ONLY == "true" ]]; then
CLEAN_OLDRUN_HRS="7777"
CLEAN_OLDFCST_HRS="7777"
fi
fi

#
# In NCO mode, the user must manually (e.g. after doing the build step)
# create the symlink "${FIXrrfs}/fix_sar" that points to EMC's FIXLAM
Expand Down
11 changes: 11 additions & 0 deletions ush/config.sh.RRFS_wcoss
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -133,6 +133,17 @@ COMINgfs=""
STMP="/gpfs/hps3/ptmp/Ming.Hu/stmp" # Path to directory STMP that mostly contains input files.
PTMP="/gpfs/hps3/ptmp/Ming.Hu/ptmp" # Path to directory STMP that mostly contains input files.

if [[ $DO_RETRO == "true" ]] ; then
CLEAN_OLDPROD_HRS="240"
CLEAN_OLDLOG_HRS="240"
CLEAN_OLDRUN_HRS="6"
CLEAN_OLDFCST_HRS="6"
if [[ $LBCS_ICS_ONLY == "true" ]]; then
CLEAN_OLDRUN_HRS="7777"
CLEAN_OLDFCST_HRS="7777"
fi
fi

#
# In NCO mode, the user must manually (e.g. after doing the build step)
# create the symlink "${FIXrrfs}/fix_sar" that points to EMC's FIXLAM
Expand Down
20 changes: 20 additions & 0 deletions ush/config_defaults.sh
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1771,3 +1771,23 @@ DO_REFL2TTEN="FALSE"
#
RADARREFL_MINS=(0 1 2 3)
RADARREFL_TIMELEVEL=(0)

#
#-----------------------------------------------------------------------
#
# Parameters for cleaning the real-time and retrospective runs.
# CLEAN_OLDPROD_HRS:
# the product under com directory from cycles older than (current cycle - this hour) will be cleaned
# CLEAN_OLDLOG_HRS
# the log files under com directory from cycles older than (current cycle - this hour) will be cleaned
# CLEAN_OLDRUN_HRS
# the run directory under tmpnwprd directory from cycles older than (current cycle - this hour) will be cleaned
# CLEAN_OLDFCST_HRS
# the fv3lam forecast netcdf files forecast run directory from cycles older than (current cycle - this hour) will be cleaned
#-----------------------------------------------------------------------
#

CLEAN_OLDPROD_HRS="72"
CLEAN_OLDLOG_HRS="48"
CLEAN_OLDRUN_HRS="72"
CLEAN_OLDFCST_HRS="24"
15 changes: 5 additions & 10 deletions ush/templates/FV3LAM_wflow.xml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -950,15 +950,7 @@ MODULES_RUN_TASK_FP script.
</dependency>
</task>

<!--
************************************************************************
************************************************************************
-->
{%- endif %}


{%- if not do_retro %}

<!--
************************************************************************
************************************************************************
Expand All @@ -975,6 +967,11 @@ MODULES_RUN_TASK_FP script.

<envar><name>GLOBAL_VAR_DEFNS_FP</name><value>&GLOBAL_VAR_DEFNS_FP;</value></envar>
<envar><name>LOGDIR</name><value>&LOGDIR;</value></envar>
<envar><name>CDATE</name><value><cyclestr>@Y@m@d@H</cyclestr></value></envar>

<dependency>
<taskdep task="&RUN_FCST_TN;"/>
</dependency>

</task>

Expand All @@ -999,6 +996,4 @@ MODULES_RUN_TASK_FP script.
</task>

{%- endif %}
{%- endif %}

</workflow>