Skip to content

Add the repair job for the reforecast#2838

Merged
EricSinsky-NOAA merged 63 commits into
NOAA-EMC:feature/gefs_reforecastfrom
HongGuan-NOAA:repair
Oct 11, 2024
Merged

Add the repair job for the reforecast#2838
EricSinsky-NOAA merged 63 commits into
NOAA-EMC:feature/gefs_reforecastfrom
HongGuan-NOAA:repair

Conversation

@HongGuan-NOAA
Copy link
Copy Markdown

@HongGuan-NOAA HongGuan-NOAA commented Aug 15, 2024

Description

This PR adds the repair task for correcting the F03 and F06 ave/acc/min/max variables.

Type of change

  • New feature (adds functionality)

Change characteristics

  • Is this a breaking change (a change in existing functionality)? NO
  • Does this change require a documentation update? NO
  • Does this change require an update to any of the following submodules? YES (If YES, please add a link to any PRs that are pending.)
    • EMC verif-global
    • GDAS
    • GFS-utils
    • GSI
    • GSI-monitor
    • GSI-utils
    • UFS-utils
    • UFS-weather-model
    • wxflow

How has this been tested?

This has been tested in WCOSS2.

Checklist

  • Any dependent changes have been merged and published
  • My code follows the style guidelines of this project
  • I have performed a self-review of my own code
  • I have commented my code, particularly in hard-to-understand areas
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • I have made corresponding changes to the documentation if necessary

@HongGuan-NOAA HongGuan-NOAA marked this pull request as draft August 15, 2024 19:33
Copy link
Copy Markdown

@github-advanced-security github-advanced-security AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ShellCheck found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.

Comment thread scripts/gefs_atmos_f0306.sh Outdated
Comment on lines +151 to +152
$exec_dir/$sorc_name >sorc_name.exe.out
cat sorc_name.exe.out
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@WalterKolczynski-NOAA, @HongGuan-NOAA has source code for this task that has not been added to this PR. Should her source code be added to gfs-utils in a similar manner as the other GEFS-specific programs (e.g. gfs-utils PR #52)?

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes. I thought I had already talked with Hong about opening a PR.

Comment thread scripts/gefs_atmos_f0306.sh Fixed
Comment thread scripts/gefs_atmos_f0306.sh Fixed
Comment thread scripts/gefs_atmos_f0306.sh Fixed
Comment thread scripts/gefs_atmos_f0306.sh Fixed
Comment thread scripts/gefs_atmos_f0306.sh Fixed
Comment thread scripts/gefs_atmos_f0306.sh Fixed
Copy link
Copy Markdown
Contributor

@EricSinsky-NOAA EricSinsky-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@HongGuan-NOAA Here are some of my suggestions.

Comment thread scripts/gefs_atmos_f0306.sh Outdated
Comment thread jobs/JGEFS_ATMOS_ACC Outdated
#
#export COMIN_master=${COMIN_master:-$COMROOT/$PSLOT/gefs.$PDY/00/mem001/model_data/atmos/master}
export COMIN_master=${COMIN_master:-${COM_ATMOS_MASTER}}
export COMIN_00and03=${HOMEgefs}/anl
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

COMIN_00and03 should not be in HOMEgefs. Since data in this directory is considered fixed (not generated in any global-workflow task), I think it can be somewhere in a common data directory outside of HOMEgefs.

Comment thread workflow/rocoto/gefs_tasks.py
Comment thread workflow/applications/gefs.py Outdated
@HongGuan-NOAA
Copy link
Copy Markdown
Author

HongGuan-NOAA commented Aug 26, 2024 via email

Copy link
Copy Markdown
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Some changes to get started. I will likely have more once things are cleaned up.

Comment thread jobs/JGEFS_ATMOS_ACC Outdated
Comment on lines +10 to +32
####################################
# obtain unique process id (pid) and make temp directory
####################################

export pid=$$
export date=${CDATE}
#yyyymmdd=`echo $date | cut -c1-8`
#echo `date` $0 `date -u` begin

####################################
# File To Log Msgs
####################################
export jlogfile=${jlogfile:-${COMROOT}/logs/jlogfiles/jlogfile.${job}.${pid}}
#echo $jlogfile

####################################
# Determine Job Output Name on System
####################################
export outid="LL${job}"
export jobid="${outid}.o${pid}"
export pgmout="OUTPUT.${pid}"
export pgmerr=errfile
#echo $outid,$jobid,$pgmout,$pgmerr
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

None of this is necessary. The jjob header takes care of what is needed, and most of the rest we don't use at all.

Suggested change
####################################
# obtain unique process id (pid) and make temp directory
####################################
export pid=$$
export date=${CDATE}
#yyyymmdd=`echo $date | cut -c1-8`
#echo `date` $0 `date -u` begin
####################################
# File To Log Msgs
####################################
export jlogfile=${jlogfile:-${COMROOT}/logs/jlogfiles/jlogfile.${job}.${pid}}
#echo $jlogfile
####################################
# Determine Job Output Name on System
####################################
export outid="LL${job}"
export jobid="${outid}.o${pid}"
export pgmout="OUTPUT.${pid}"
export pgmerr=errfile
#echo $outid,$jobid,$pgmout,$pgmerr

Comment thread jobs/JGEFS_ATMOS_ACC Outdated
####################################
# Specify Execution Areas
####################################
export HOMEgefs=${HOMEgfs}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please just use HOMEgfs instead of defining HOMEgefs.

Comment thread jobs/JGEFS_ATMOS_ACC Outdated
# Specify Execution Areas
####################################
export HOMEgefs=${HOMEgfs}
export EXECacc=${EXECacc:-${HOMEgefs}/exec}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We already have EXECgfs for this.

Comment thread jobs/JGEFS_ATMOS_ACC Outdated
####################################
export HOMEgefs=${HOMEgfs}
export EXECacc=${EXECacc:-${HOMEgefs}/exec}
export SORCacc=${SORCacc:-${HOMEgefs}/sorc/gefs_postacc.fd}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Why is this necessary

Comment thread jobs/JGEFS_ATMOS_ACC Outdated
export HOMEgefs=${HOMEgfs}
export EXECacc=${EXECacc:-${HOMEgefs}/exec}
export SORCacc=${SORCacc:-${HOMEgefs}/sorc/gefs_postacc.fd}
export SCRIPTSens_acc=${HOMEgefs}/scripts
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Use SCRgfs

Comment thread scripts/gefs_atmos_f0306.sh Outdated
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please fix the indentation in this script.

Also, there is quite a bit of repetition. Consider consolidating it with loops.

Comment thread scripts/gefs_atmos_f0306.sh Outdated

rm tmp
else
echo "${infile} does not exist"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Shouldn't this exit with an error?

Comment thread workflow/applications/gefs.py Outdated
Returns the config_files that are involved in gefs
"""
configs = ['stage_ic', 'fcst', 'atmos_products', 'arch']
configs = ['stage_ic', 'fcst', 'repairf0306', 'atmos_products', 'arch']
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Can we call the new job (and associated scripts) repair_replay_acc or something similar?

Comment on lines +172 to +174
def atmos_prod(self):
return self._atmosoceaniceprod('atmos')

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This method is already defined.

Suggested change
def atmos_prod(self):
return self._atmosoceaniceprod('atmos')

Comment thread workflow/rocoto/gefs_tasks.py Outdated
Comment on lines +179 to +186
products_dict = {'atmos': {'config': 'atmos_products',
'history_path_tmpl': 'COM_ATMOS_MASTER_TMPL',
'history_file_tmpl': f'{self.run}.t@Hz.master.grb2f006'}}
component_dict = products_dict['atmos']
config = component_dict['config']
history_path_tmpl = component_dict['history_path_tmpl']
history_file_tmpl = component_dict['history_file_tmpl']
history_path = self._template_to_rocoto_cycstring(self._base[history_path_tmpl], {'MEMDIR': 'mem#member#'})
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
products_dict = {'atmos': {'config': 'atmos_products',
'history_path_tmpl': 'COM_ATMOS_MASTER_TMPL',
'history_file_tmpl': f'{self.run}.t@Hz.master.grb2f006'}}
component_dict = products_dict['atmos']
config = component_dict['config']
history_path_tmpl = component_dict['history_path_tmpl']
history_file_tmpl = component_dict['history_file_tmpl']
history_path = self._template_to_rocoto_cycstring(self._base[history_path_tmpl], {'MEMDIR': 'mem#member#'})
history_path = self._template_to_rocoto_cycstring(self._base["COM_ATMOS_HISTORY_TMPL"], {'MEMDIR': 'mem#member#'})

Comment thread workflow/rocoto/gefs_tasks.py Outdated
history_file_tmpl = component_dict['history_file_tmpl']
history_path = self._template_to_rocoto_cycstring(self._base[history_path_tmpl], {'MEMDIR': 'mem#member#'})

data = f'{history_path}/{history_file_tmpl}'
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
data = f'{history_path}/{history_file_tmpl}'
data = f'{history_path}/{self.run}.t@Hz.master.grb2f006'

@EricSinsky-NOAA EricSinsky-NOAA changed the base branch from develop to feature/gefs_reforecast September 4, 2024 19:54
@EricSinsky-NOAA
Copy link
Copy Markdown
Contributor

The target has been changed from develop to feature/gefs_reforecast.

Copy link
Copy Markdown
Contributor

@WalterKolczynski-NOAA WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note: if #2981 gets merged first, the task names here will need to be updated.

Comment thread jobs/JGEFS_ATMOS_ACC Outdated
Comment on lines +5 to +6
# 8/14/2024, Hong Guan Scripts to repair F0306 for GEFSv13 reforecasts
####################################################################
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Version control records the history of the file, it does not need to be done here.

Suggested change
# 8/14/2024, Hong Guan Scripts to repair F0306 for GEFSv13 reforecasts
####################################################################

Comment thread jobs/JGEFS_ATMOS_ACC Outdated
export EXECacc=${EXECacc:-${HOMEgfs}/exec}

# Construct COM variables from templates
YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COMIN_ATMOS_MASTER:COM_ATMOS_MASTER_TMPL
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Since this is writing to the master directory, we need both an in and an out for consistency, even if they are the same place.

Suggested change
YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COMIN_ATMOS_MASTER:COM_ATMOS_MASTER_TMPL
YMD=${PDY} HH=${cyc} declare_from_tmpl -rx \
COMIN_ATMOS_MASTER:COM_ATMOS_MASTER_TMPL
COMOUT_ATMOS_MASTER:COM_ATMOS_MASTER_TMPL

Comment thread jobs/JGEFS_ATMOS_ACC Outdated
####################################
# Specify Execution Areas
####################################
export EXECacc=${EXECacc:-${HOMEgfs}/exec}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Please use EXECgfs where you are currently using EXECacc.

Suggested change
export EXECacc=${EXECacc:-${HOMEgfs}/exec}

Comment thread jobs/JGEFS_ATMOS_ACC Outdated
########################################################
# Execute the acc script.
"${SCRgfs}/exgefs_atmos_repair_replay.sh"

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
export err=$?; err_chk

Comment on lines +2 to +3


Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change

Comment thread scripts/exgefs_atmos_repair_replay.sh Outdated

# cd $DATA

"${exec_dir}/${sorc_name}" >sorc_name.exe.out
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
"${exec_dir}/${sorc_name}" >sorc_name.exe.out
"${EXECgfs}/${sorc_name}" > sorc_name.exe.out

Comment thread scripts/exgefs_atmos_repair_replay.sh Outdated
Comment on lines +75 to +76
export exec_dir=${HOMEgfs}/exec
# export sorc_dir=${SORCacc}
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
export exec_dir=${HOMEgfs}/exec
# export sorc_dir=${SORCacc}

Comment thread scripts/exgefs_atmos_repair_replay.sh Outdated
Comment on lines +96 to +97
#output f03
export varlist=${varlist_masteracc03} # Parameter table for f06 acc/ave/min/max variables
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
#output f03
export varlist=${varlist_masteracc03} # Parameter table for f06 acc/ave/min/max variables
varlist=${varlist_masteracc03} # Parameter table for f06 acc/ave/min/max variables

Comment thread scripts/exgefs_atmos_repair_replay.sh Outdated
Comment on lines +107 to +108
mv "${COMIN_ATMOS_MASTER}/gefs.t00z.master.grb2f003" "${COMIN_ATMOS_MASTER}/gefs.t00z.master.grb2f003_org"
cat out3.grb2 TSNOWP1.dat gefs.t00z.pgrb2af003 > "${COMIN_ATMOS_MASTER}/gefs.t00z.master.grb2f003"
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
mv "${COMIN_ATMOS_MASTER}/gefs.t00z.master.grb2f003" "${COMIN_ATMOS_MASTER}/gefs.t00z.master.grb2f003_org"
cat out3.grb2 TSNOWP1.dat gefs.t00z.pgrb2af003 > "${COMIN_ATMOS_MASTER}/gefs.t00z.master.grb2f003"
cat out3.grb2 TSNOWP1.dat gefs.t00z.pgrb2af003 > gefs.t00z.master.grb2f003
${NCP} "${COMIN_ATMOS_MASTER}/gefs.t00z.master.grb2f003" "${COMOUT_ATMOS_MASTER}/gefs.t00z.master.grb2f003_org"
${NCP} gefs.t00z.master.grb2f003 "${COMOUT_ATMOS_MASTER}/gefs.t00z.master.grb2f003"

Comment thread workflow/rocoto/gefs_tasks.py Outdated
def repair_replay(self):

deps = []
# history_path = self._template_to_rocoto_cycstring(self._base["COM_ATMOS_HISTORY_TMPL"], {'MEMDIR': 'mem#member#'})
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# history_path = self._template_to_rocoto_cycstring(self._base["COM_ATMOS_HISTORY_TMPL"], {'MEMDIR': 'mem#member#'})

Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
Comment thread scripts/exgefs_atmos_repair_replay.sh Fixed
@WalterKolczynski-NOAA
Copy link
Copy Markdown
Contributor

WalterKolczynski-NOAA commented Oct 9, 2024

@HongGuan-NOAA Time for a different approach. Get rid of all the || ... after the pipe chains and just add the following at the top of the script:

# shellcheck disable=SC2312

@HongGuan-NOAA
Copy link
Copy Markdown
Author

HongGuan-NOAA commented Oct 9, 2024 via email

Comment thread workflow/applications/applications.py
Copy link
Copy Markdown
Contributor

@EricSinsky-NOAA EricSinsky-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The repair job has succeeded in my tests. Approving this to be merged to gefs_reforecast. Code maintenance-related issues can be addressed in future PRs.

@EricSinsky-NOAA
Copy link
Copy Markdown
Contributor

@WalterKolczynski-NOAA After speaking with Hong, she has finished going through the review comments. I can merge this to gefs_reforecast if you think this PR is ready. Thanks.

@HongGuan-NOAA
Copy link
Copy Markdown
Author

HongGuan-NOAA commented Oct 11, 2024 via email

@EricSinsky-NOAA EricSinsky-NOAA merged commit abf7467 into NOAA-EMC:feature/gefs_reforecast Oct 11, 2024
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants