Add the repair job for the reforecast#2838
Conversation
There was a problem hiding this comment.
ShellCheck found more than 20 potential problems in the proposed changes. Check the Files changed tab for more details.
| $exec_dir/$sorc_name >sorc_name.exe.out | ||
| cat sorc_name.exe.out |
There was a problem hiding this comment.
@WalterKolczynski-NOAA, @HongGuan-NOAA has source code for this task that has not been added to this PR. Should her source code be added to gfs-utils in a similar manner as the other GEFS-specific programs (e.g. gfs-utils PR #52)?
There was a problem hiding this comment.
Yes. I thought I had already talked with Hong about opening a PR.
This reverts commit e447511.
EricSinsky-NOAA
left a comment
There was a problem hiding this comment.
@HongGuan-NOAA Here are some of my suggestions.
| # | ||
| #export COMIN_master=${COMIN_master:-$COMROOT/$PSLOT/gefs.$PDY/00/mem001/model_data/atmos/master} | ||
| export COMIN_master=${COMIN_master:-${COM_ATMOS_MASTER}} | ||
| export COMIN_00and03=${HOMEgefs}/anl |
There was a problem hiding this comment.
COMIN_00and03 should not be in HOMEgefs. Since data in this directory is considered fixed (not generated in any global-workflow task), I think it can be somewhere in a common data directory outside of HOMEgefs.
|
I will take a look. Thanks for looking into this. -Hong
…On Mon, Aug 26, 2024 at 8:29 AM Eric Sinsky - NOAA ***@***.***> wrote:
***@***.**** commented on this pull request.
@HongGuan-NOAA <https://github.com/HongGuan-NOAA> Here are some of my
suggestions.
------------------------------
In scripts/gefs_atmos_f0306.sh
<#2838 (comment)>
:
> @@ -0,0 +1,192 @@
+#!/bin/ksh
This script should start like this. A brief description of this script
should be added as well.
⬇️ Suggested change
-#!/bin/ksh
+#! /usr/bin/env bash
+
+source "${USHgfs}/preamble.sh"
------------------------------
In jobs/JGEFS_ATMOS_ACC
<#2838 (comment)>
:
> +
+export PARMgefs=${HOMEgefs}/rocoto/parm
+echo "${PARMgefs}"
+
+# Construct COM variables from templates
+#YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COM_ATMOS_ANALYSIS COM_ATMOS_HISTORY COM_ATMOS_MASTER
+ YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COM_ATMOS_MASTER
+
+
+##############################
+# Run setpdy and initialize PDY variables
+##############################
+#
+#export COMIN_master=${COMIN_master:-$COMROOT/$PSLOT/gefs.$PDY/00/mem001/model_data/atmos/master}
+export COMIN_master=${COMIN_master:-${COM_ATMOS_MASTER}}
+export COMIN_00and03=${HOMEgefs}/anl
COMIN_00and03 should not be in HOMEgefs. Since data in this directory is
considered fixed (not generated in any global-workflow task), I think it
can be somewhere in a common data directory outside of HOMEgefs.
------------------------------
In workflow/rocoto/gefs_tasks.py
<#2838 (comment)>
:
> +
+ def repairf0306(self):
+
+ deps = []
+
+ products_dict = {'atmos': {'config': 'atmos_products',
+ 'history_path_tmpl': 'COM_ATMOS_MASTER_TMPL',
+ 'history_file_tmpl': ***@***.***'}}
+ component_dict = products_dict['atmos']
+ config = component_dict['config']
+ history_path_tmpl = component_dict['history_path_tmpl']
+ history_file_tmpl = component_dict['history_file_tmpl']
+ history_path = self._template_to_rocoto_cycstring(self._base[history_path_tmpl], {'MEMDIR': 'mem#member#'})
+
+ data = f'{history_path}/{history_file_tmpl}'
+ dep_dict = {'type': 'data', 'data': data}
Since the repair job depends on f006 master data from the forecast job, I
think the dependencies for the f003 and f006 atmos_prod jobs should also be
modified (for replay cases). The f003 and f006 atmos_prod jobs should
depend on the successful completion of the repair job.
------------------------------
In workflow/applications/gefs.py
<#2838 (comment)>
:
> @@ -59,6 +59,8 @@ def get_task_names(self):
if self.nens > 0:
tasks += ['efcs']
+ tasks += ['repairf0306']
I think this task should be added to the workflow only for replay cases.
This task should probably be conditioned. I suggest reviewing the
extractvars task to see how a task can be made optional in the workflow.
—
Reply to this email directly, view it on GitHub
<#2838 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AVLBRUQPW5P2PUBZPVK3LQTZTMNRFAVCNFSM6AAAAABMSYUJ22VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDENRQGUYTINBSHE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
WalterKolczynski-NOAA
left a comment
There was a problem hiding this comment.
Some changes to get started. I will likely have more once things are cleaned up.
| #################################### | ||
| # obtain unique process id (pid) and make temp directory | ||
| #################################### | ||
|
|
||
| export pid=$$ | ||
| export date=${CDATE} | ||
| #yyyymmdd=`echo $date | cut -c1-8` | ||
| #echo `date` $0 `date -u` begin | ||
|
|
||
| #################################### | ||
| # File To Log Msgs | ||
| #################################### | ||
| export jlogfile=${jlogfile:-${COMROOT}/logs/jlogfiles/jlogfile.${job}.${pid}} | ||
| #echo $jlogfile | ||
|
|
||
| #################################### | ||
| # Determine Job Output Name on System | ||
| #################################### | ||
| export outid="LL${job}" | ||
| export jobid="${outid}.o${pid}" | ||
| export pgmout="OUTPUT.${pid}" | ||
| export pgmerr=errfile | ||
| #echo $outid,$jobid,$pgmout,$pgmerr |
There was a problem hiding this comment.
None of this is necessary. The jjob header takes care of what is needed, and most of the rest we don't use at all.
| #################################### | |
| # obtain unique process id (pid) and make temp directory | |
| #################################### | |
| export pid=$$ | |
| export date=${CDATE} | |
| #yyyymmdd=`echo $date | cut -c1-8` | |
| #echo `date` $0 `date -u` begin | |
| #################################### | |
| # File To Log Msgs | |
| #################################### | |
| export jlogfile=${jlogfile:-${COMROOT}/logs/jlogfiles/jlogfile.${job}.${pid}} | |
| #echo $jlogfile | |
| #################################### | |
| # Determine Job Output Name on System | |
| #################################### | |
| export outid="LL${job}" | |
| export jobid="${outid}.o${pid}" | |
| export pgmout="OUTPUT.${pid}" | |
| export pgmerr=errfile | |
| #echo $outid,$jobid,$pgmout,$pgmerr |
| #################################### | ||
| # Specify Execution Areas | ||
| #################################### | ||
| export HOMEgefs=${HOMEgfs} |
There was a problem hiding this comment.
Please just use HOMEgfs instead of defining HOMEgefs.
| # Specify Execution Areas | ||
| #################################### | ||
| export HOMEgefs=${HOMEgfs} | ||
| export EXECacc=${EXECacc:-${HOMEgefs}/exec} |
There was a problem hiding this comment.
We already have EXECgfs for this.
| #################################### | ||
| export HOMEgefs=${HOMEgfs} | ||
| export EXECacc=${EXECacc:-${HOMEgefs}/exec} | ||
| export SORCacc=${SORCacc:-${HOMEgefs}/sorc/gefs_postacc.fd} |
There was a problem hiding this comment.
Why is this necessary
| export HOMEgefs=${HOMEgfs} | ||
| export EXECacc=${EXECacc:-${HOMEgefs}/exec} | ||
| export SORCacc=${SORCacc:-${HOMEgefs}/sorc/gefs_postacc.fd} | ||
| export SCRIPTSens_acc=${HOMEgefs}/scripts |
There was a problem hiding this comment.
Please fix the indentation in this script.
Also, there is quite a bit of repetition. Consider consolidating it with loops.
|
|
||
| rm tmp | ||
| else | ||
| echo "${infile} does not exist" |
There was a problem hiding this comment.
Shouldn't this exit with an error?
| Returns the config_files that are involved in gefs | ||
| """ | ||
| configs = ['stage_ic', 'fcst', 'atmos_products', 'arch'] | ||
| configs = ['stage_ic', 'fcst', 'repairf0306', 'atmos_products', 'arch'] |
There was a problem hiding this comment.
Can we call the new job (and associated scripts) repair_replay_acc or something similar?
| def atmos_prod(self): | ||
| return self._atmosoceaniceprod('atmos') | ||
|
|
There was a problem hiding this comment.
This method is already defined.
| def atmos_prod(self): | |
| return self._atmosoceaniceprod('atmos') |
| products_dict = {'atmos': {'config': 'atmos_products', | ||
| 'history_path_tmpl': 'COM_ATMOS_MASTER_TMPL', | ||
| 'history_file_tmpl': f'{self.run}.t@Hz.master.grb2f006'}} | ||
| component_dict = products_dict['atmos'] | ||
| config = component_dict['config'] | ||
| history_path_tmpl = component_dict['history_path_tmpl'] | ||
| history_file_tmpl = component_dict['history_file_tmpl'] | ||
| history_path = self._template_to_rocoto_cycstring(self._base[history_path_tmpl], {'MEMDIR': 'mem#member#'}) |
There was a problem hiding this comment.
| products_dict = {'atmos': {'config': 'atmos_products', | |
| 'history_path_tmpl': 'COM_ATMOS_MASTER_TMPL', | |
| 'history_file_tmpl': f'{self.run}.t@Hz.master.grb2f006'}} | |
| component_dict = products_dict['atmos'] | |
| config = component_dict['config'] | |
| history_path_tmpl = component_dict['history_path_tmpl'] | |
| history_file_tmpl = component_dict['history_file_tmpl'] | |
| history_path = self._template_to_rocoto_cycstring(self._base[history_path_tmpl], {'MEMDIR': 'mem#member#'}) | |
| history_path = self._template_to_rocoto_cycstring(self._base["COM_ATMOS_HISTORY_TMPL"], {'MEMDIR': 'mem#member#'}) |
| history_file_tmpl = component_dict['history_file_tmpl'] | ||
| history_path = self._template_to_rocoto_cycstring(self._base[history_path_tmpl], {'MEMDIR': 'mem#member#'}) | ||
|
|
||
| data = f'{history_path}/{history_file_tmpl}' |
There was a problem hiding this comment.
| data = f'{history_path}/{history_file_tmpl}' | |
| data = f'{history_path}/{self.run}.t@Hz.master.grb2f006' |
|
The target has been changed from |
WalterKolczynski-NOAA
left a comment
There was a problem hiding this comment.
Note: if #2981 gets merged first, the task names here will need to be updated.
| # 8/14/2024, Hong Guan Scripts to repair F0306 for GEFSv13 reforecasts | ||
| #################################################################### |
There was a problem hiding this comment.
Version control records the history of the file, it does not need to be done here.
| # 8/14/2024, Hong Guan Scripts to repair F0306 for GEFSv13 reforecasts | |
| #################################################################### |
| export EXECacc=${EXECacc:-${HOMEgfs}/exec} | ||
|
|
||
| # Construct COM variables from templates | ||
| YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COMIN_ATMOS_MASTER:COM_ATMOS_MASTER_TMPL |
There was a problem hiding this comment.
Since this is writing to the master directory, we need both an in and an out for consistency, even if they are the same place.
| YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COMIN_ATMOS_MASTER:COM_ATMOS_MASTER_TMPL | |
| YMD=${PDY} HH=${cyc} declare_from_tmpl -rx \ | |
| COMIN_ATMOS_MASTER:COM_ATMOS_MASTER_TMPL | |
| COMOUT_ATMOS_MASTER:COM_ATMOS_MASTER_TMPL |
| #################################### | ||
| # Specify Execution Areas | ||
| #################################### | ||
| export EXECacc=${EXECacc:-${HOMEgfs}/exec} |
There was a problem hiding this comment.
Please use EXECgfs where you are currently using EXECacc.
| export EXECacc=${EXECacc:-${HOMEgfs}/exec} |
| ######################################################## | ||
| # Execute the acc script. | ||
| "${SCRgfs}/exgefs_atmos_repair_replay.sh" | ||
|
|
There was a problem hiding this comment.
| export err=$?; err_chk | |
|
|
||
|
|
There was a problem hiding this comment.
|
|
||
| # cd $DATA | ||
|
|
||
| "${exec_dir}/${sorc_name}" >sorc_name.exe.out |
There was a problem hiding this comment.
| "${exec_dir}/${sorc_name}" >sorc_name.exe.out | |
| "${EXECgfs}/${sorc_name}" > sorc_name.exe.out |
| export exec_dir=${HOMEgfs}/exec | ||
| # export sorc_dir=${SORCacc} |
There was a problem hiding this comment.
| export exec_dir=${HOMEgfs}/exec | |
| # export sorc_dir=${SORCacc} |
| #output f03 | ||
| export varlist=${varlist_masteracc03} # Parameter table for f06 acc/ave/min/max variables |
There was a problem hiding this comment.
| #output f03 | |
| export varlist=${varlist_masteracc03} # Parameter table for f06 acc/ave/min/max variables | |
| varlist=${varlist_masteracc03} # Parameter table for f06 acc/ave/min/max variables |
| mv "${COMIN_ATMOS_MASTER}/gefs.t00z.master.grb2f003" "${COMIN_ATMOS_MASTER}/gefs.t00z.master.grb2f003_org" | ||
| cat out3.grb2 TSNOWP1.dat gefs.t00z.pgrb2af003 > "${COMIN_ATMOS_MASTER}/gefs.t00z.master.grb2f003" |
There was a problem hiding this comment.
| mv "${COMIN_ATMOS_MASTER}/gefs.t00z.master.grb2f003" "${COMIN_ATMOS_MASTER}/gefs.t00z.master.grb2f003_org" | |
| cat out3.grb2 TSNOWP1.dat gefs.t00z.pgrb2af003 > "${COMIN_ATMOS_MASTER}/gefs.t00z.master.grb2f003" | |
| cat out3.grb2 TSNOWP1.dat gefs.t00z.pgrb2af003 > gefs.t00z.master.grb2f003 | |
| ${NCP} "${COMIN_ATMOS_MASTER}/gefs.t00z.master.grb2f003" "${COMOUT_ATMOS_MASTER}/gefs.t00z.master.grb2f003_org" | |
| ${NCP} gefs.t00z.master.grb2f003 "${COMOUT_ATMOS_MASTER}/gefs.t00z.master.grb2f003" |
| def repair_replay(self): | ||
|
|
||
| deps = [] | ||
| # history_path = self._template_to_rocoto_cycstring(self._base["COM_ATMOS_HISTORY_TMPL"], {'MEMDIR': 'mem#member#'}) |
There was a problem hiding this comment.
| # history_path = self._template_to_rocoto_cycstring(self._base["COM_ATMOS_HISTORY_TMPL"], {'MEMDIR': 'mem#member#'}) |
|
@HongGuan-NOAA Time for a different approach. Get rid of all the |
|
I added the following line. It works now. Thanks!
# shellcheck disable=SC2312
…On Wed, Oct 9, 2024 at 1:52 PM Walter Kolczynski - NOAA < ***@***.***> wrote:
@HongGuan-NOAA <https://github.com/HongGuan-NOAA> Time for a different
approach. Get rid of all the || ... and just add the following at the top
of the script:
# shellcheck disable=SC2312
—
Reply to this email directly, view it on GitHub
<#2838 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AVLBRUXCWOPZTH3UECBDLY3Z2VUN5AVCNFSM6AAAAABMSYUJ22VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDIMBSHEZTOMZWHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
EricSinsky-NOAA
left a comment
There was a problem hiding this comment.
The repair job has succeeded in my tests. Approving this to be merged to gefs_reforecast. Code maintenance-related issues can be addressed in future PRs.
|
@WalterKolczynski-NOAA After speaking with Hong, she has finished going through the review comments. I can merge this to |
|
Thanks for this effort.
…On Fri, Oct 11, 2024 at 11:09 AM Eric Sinsky - NOAA < ***@***.***> wrote:
***@***.**** approved this pull request.
The repair job has succeeded in my tests. Approving this to be merged to
gefs_reforecast. Code maintenance-related issues can be addressed in
future PRs.
—
Reply to this email directly, view it on GitHub
<#2838 (review)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AVLBRUSWLT37BO5M6OFBVA3Z27SZJAVCNFSM6AAAAABMSYUJ22VHI2DSMVQWIX3LMV43YUDVNRWFEZLROVSXG5CSMV3GSZLXHMZDGNRTGAZTOMBSHE>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Description
This PR adds the repair task for correcting the F03 and F06 ave/acc/min/max variables.
Type of change
Change characteristics
How has this been tested?
This has been tested in WCOSS2.
Checklist