Skip to content

Stage bias files for UFSDA aerosols #1370

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
WalterKolczynski-NOAA merged 27 commits into from
Apr 12, 2023
Merged

Stage bias files for UFSDA aerosols #1370

Show file tree
Hide file tree
Changes from 26 commits
Commits
Show all changes
27 commits
Select commit Hold shift + click to select a range
c4d208e
changes related to UFS-DA aero (#1313)
RussTreadon-NOAA Mar 3, 2023
f196d3a
clean up analysis.py based on reviewer suggestions (#1313)
RussTreadon-NOAA Mar 4, 2023
5e8cb1c
replace obfile loop with replace in get_bias_dict (#1313)
RussTreadon-NOAA Mar 6, 2023
34650d4
Merge branch 'NOAA-EMC:develop' into feature/update_jediaero
RussTreadon-NOAA Mar 7, 2023
5182c50
Merge branch 'NOAA-EMC:develop' into feature/update_jediaero
RussTreadon-NOAA Mar 8, 2023
0685678
Merge branch 'NOAA-EMC:develop' into feature/update_jediaero
RussTreadon-NOAA Mar 10, 2023
3200e17
Merge branch 'NOAA-EMC:develop' into feature/update_jediaero
RussTreadon-NOAA Mar 13, 2023
74efd03
Merge branch 'NOAA-EMC:develop' into feature/update_jediaero
RussTreadon-NOAA Mar 15, 2023
b1cc1b6
Merge branch 'NOAA-EMC:develop' into feature/update_jediaero
RussTreadon-NOAA Mar 17, 2023
e97d43d
Merge branch 'NOAA-EMC:develop' into feature/update_jediaero
RussTreadon-NOAA Mar 21, 2023
f96949d
Merge branch 'NOAA-EMC:develop' into feature/update_jediaero
RussTreadon-NOAA Mar 22, 2023
c35cde7
Merge branch 'NOAA-EMC:develop' into feature/update_jediaero
RussTreadon-NOAA Mar 22, 2023
cebcbcd
Merge branch 'NOAA-EMC:develop' into feature/update_jediaero
RussTreadon-NOAA Mar 23, 2023
07e23ed
Merge branch 'NOAA-EMC:develop' into feature/update_jediaero
RussTreadon-NOAA Mar 24, 2023
aa858ec
Merge branch 'develop' into feature/update_jediaero
RussTreadon-NOAA Mar 27, 2023
3e7a509
rename obs_dict as bias_dict for clarity in analysis.py (#1313)
RussTreadon-NOAA Mar 27, 2023
7556a11
remove blank line from analysis.py (#1313)
RussTreadon-NOAA Mar 27, 2023
fb3204f
Merge branch 'NOAA-EMC:develop' into feature/update_jediaero
RussTreadon-NOAA Mar 28, 2023
76e22e7
Merge branch 'NOAA-EMC:develop' into feature/update_jediaero
RussTreadon-NOAA Mar 28, 2023
a022aef
update aero_crtm_coef.yaml to new g-w format (#1313)
RussTreadon-NOAA Mar 28, 2023
a789094
Merge branch 'NOAA-EMC:develop' into feature/update_jediaero
RussTreadon-NOAA Mar 29, 2023
82cf879
replace YAMLFile with parse_j2yaml in analysis.py (#1313)
RussTreadon-NOAA Mar 30, 2023
8d166d6
Merge branch 'NOAA-EMC:develop' into feature/update_jediaero
RussTreadon-NOAA Mar 31, 2023
9b068f3
Merge branch 'NOAA-EMC:develop' into feature/update_jediaero
RussTreadon-NOAA Mar 31, 2023
f369a36
Merge branch 'NOAA-EMC:develop' into feature/update_jediaero
RussTreadon-NOAA Apr 4, 2023
b023df5
Merge branch 'NOAA-EMC:develop' into feature/update_jediaero
RussTreadon-NOAA Apr 4, 2023
ef5da49
streamline radiance bias correction file processing based on reviewer…
RussTreadon-NOAA Apr 12, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
10 changes: 9 additions & 1 deletion env/JET.env
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4,7 +4,7 @@ if [[ $# -ne 1 ]]; then

echo "Must specify an input argument to set runtime environment variables!"
echo "argument can be any one of the following:"
echo "atmanalrun atmensanalrun"
echo "atmanalrun atmensanalrun aeroanlrun"
echo "anal sfcanl fcst post vrfy metp"
echo "eobs eupd ecen efcs epos"
echo "postsnd awips gempak"
Expand Down Expand Up @@ -59,6 +59,14 @@ elif [[ "${step}" = "atmensanalrun" ]]; then
[[ ${NTHREADS_ATMENSANAL} -gt ${nth_max} ]] && export NTHREADS_ATMENSANAL=${nth_max}
export APRUN_ATMENSANAL="${launcher} ${npe_atmensanalrun}"

elif [[ "${step}" = "aeroanlrun" ]]; then

nth_max=$((npe_node_max / npe_node_aeroanlrun))

export NTHREADS_AEROANL=${nth_aeroanlrun:-${nth_max}}
[[ ${NTHREADS_AEROANL} -gt ${nth_max} ]] && export NTHREADS_AEROANL=${nth_max}
export APRUN_AEROANL="${launcher} ${npe_aeroanlrun}"

elif [[ "${step}" = "anal" ]]; then

nth_max=$((npe_node_max / npe_node_anal))
Expand Down
2 changes: 1 addition & 1 deletion parm/config/config.aeroanl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,6 +19,6 @@ export io_layout_x=@IO_LAYOUT_X@
export io_layout_y=@IO_LAYOUT_Y@

export JEDIVAREXE=${HOMEgfs}/exec/fv3jedi_var.x
export CRTM_VER="2.3.0"
export crtm_VERSION="2.3.0"

echo "END: config.aeroanl"
5 changes: 3 additions & 2 deletions parm/config/config.resources
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -60,11 +60,12 @@ if [ ${step} = "prep" ]; then
export memory_prep="40G"
fi

elif [ ${step} = "aerosol_init" ]; then
elif [[ "${step}" = "aerosol_init" ]]; then
Comment thread
RussTreadon-NOAA marked this conversation as resolved.
export wtime_aerosol_init="00:05:00"
export npe_aerosol_init=1
export nth_aerosol_init=1
export npe_node_aerosol_init=$(echo "${npe_node_max} / ${nth_aerosol_init}" | bc)
npe_node_aerosol_init=$(echo "${npe_node_max} / ${nth_aerosol_init}" | bc)
export npe_node_aerosol_init
export NTASKS=${npe_aerosol_init}
export memory_aerosol_init="6G"

Expand Down
32 changes: 11 additions & 21 deletions parm/parm_gdas/aero_crtm_coeff.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,23 +1,13 @@
mkdir:
- !ENV ${DATA}/crtm/
- $(DATA)/crtm/
copy:
- - !ENV ${FV3JEDI_FIX}/crtm/${CRTM_VER}/AerosolCoeff.bin
- !ENV ${DATA}/crtm/AerosolCoeff.bin
- - !ENV ${FV3JEDI_FIX}/crtm/${CRTM_VER}/CloudCoeff.bin
- !ENV ${DATA}/crtm/CloudCoeff.bin
- - !ENV ${FV3JEDI_FIX}/crtm/${CRTM_VER}/v.viirs-m_npp.SpcCoeff.bin
- !ENV ${DATA}/crtm/v.viirs-m_npp.SpcCoeff.bin
- - !ENV ${FV3JEDI_FIX}/crtm/${CRTM_VER}/v.viirs-m_npp.TauCoeff.bin
- !ENV ${DATA}/crtm/v.viirs-m_npp.TauCoeff.bin
- - !ENV ${FV3JEDI_FIX}/crtm/${CRTM_VER}/v.viirs-m_j1.SpcCoeff.bin
- !ENV ${DATA}/crtm/v.viirs-m_j1.SpcCoeff.bin
- - !ENV ${FV3JEDI_FIX}/crtm/${CRTM_VER}/v.viirs-m_j1.TauCoeff.bin
- !ENV ${DATA}/crtm/v.viirs-m_j1.TauCoeff.bin
- - !ENV ${FV3JEDI_FIX}/crtm/${CRTM_VER}/NPOESS.VISice.EmisCoeff.bin
- !ENV ${DATA}/crtm/NPOESS.VISice.EmisCoeff.bin
- - !ENV ${FV3JEDI_FIX}/crtm/${CRTM_VER}/NPOESS.VISland.EmisCoeff.bin
- !ENV ${DATA}/crtm/NPOESS.VISland.EmisCoeff.bin
- - !ENV ${FV3JEDI_FIX}/crtm/${CRTM_VER}/NPOESS.VISsnow.EmisCoeff.bin
- !ENV ${DATA}/crtm/NPOESS.VISsnow.EmisCoeff.bin
- - !ENV ${FV3JEDI_FIX}/crtm/${CRTM_VER}/NPOESS.VISwater.EmisCoeff.bin
- !ENV ${DATA}/crtm/NPOESS.VISwater.EmisCoeff.bin
- [$(FV3JEDI_FIX)/crtm/$(crtm_VERSION)/AerosolCoeff.bin, $(DATA)crtm/]
- [$(FV3JEDI_FIX)/crtm/$(crtm_VERSION)/CloudCoeff.bin, $(DATA)/crtm/]
- [$(FV3JEDI_FIX)/crtm/$(crtm_VERSION)/v.viirs-m_npp.SpcCoeff.bin, $(DATA)/crtm/]
- [$(FV3JEDI_FIX)/crtm/$(crtm_VERSION)/v.viirs-m_npp.TauCoeff.bin, $(DATA)/crtm/]
- [$(FV3JEDI_FIX)/crtm/$(crtm_VERSION)/v.viirs-m_j1.SpcCoeff.bin, $(DATA)/crtm/]
- [$(FV3JEDI_FIX)/crtm/$(crtm_VERSION)/v.viirs-m_j1.TauCoeff.bin, $(DATA)/crtm/]
- [$(FV3JEDI_FIX)/crtm/$(crtm_VERSION)/NPOESS.VISice.EmisCoeff.bin, $(DATA)/crtm/]
- [$(FV3JEDI_FIX)/crtm/$(crtm_VERSION)/NPOESS.VISland.EmisCoeff.bin, $(DATA)/crtm/]
- [$(FV3JEDI_FIX)/crtm/$(crtm_VERSION)/NPOESS.VISsnow.EmisCoeff.bin, $(DATA)/crtm/]
- [$(FV3JEDI_FIX)/crtm/$(crtm_VERSION)/NPOESS.VISwater.EmisCoeff.bin, $(DATA)/crtm/]
47 changes: 47 additions & 0 deletions ush/python/pygfs/task/analysis.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -32,6 +32,10 @@ def initialize(self) -> None:
obs_dict = self.get_obs_dict()
FileHandler(obs_dict).sync()

# some analyses need to stage bias corrections
bias_dict = self.get_bias_dict()
FileHandler(bias_dict).sync()

@logit(logger)
def get_obs_dict(self: Task) -> Dict[str, Any]:
"""Compile a dictionary of observation files to copy
Expand Down Expand Up @@ -64,6 +68,49 @@ def get_obs_dict(self: Task) -> Dict[str, Any]:
}
return obs_dict

@logit(logger)
def get_bias_dict(self: Task) -> Dict[str, Any]:
"""Compile a dictionary of observation files to copy

This method uses the OBS_LIST configuration variable to generate a dictionary
from a list of YAML files that specify what observation bias correction files
are to be copied to the run directory from the observation input directory

Parameters
----------

Returns
----------
bias_dict: Dict
a dictionary containing the list of observation bias files to copy for FileHandler
"""
logger.debug(f"OBS_LIST: {self.task_config['OBS_LIST']}")
obs_list_config = parse_j2yaml(self.task_config["OBS_LIST"], self.task_config)
logger.debug(f"obs_list_config: {obs_list_config}")
# get observers from master dictionary
observers = obs_list_config['observers']
copylist = []
for ob in observers:
if 'obs bias' in ob.keys():
obfile = ob['obs bias']['input file']
basename = os.path.basename(obfile)
copylist.append([os.path.join(self.task_config.comin_ges_atm, basename), obfile])

obfile = obfile.replace('satbias', 'satbias_cov')
basename = os.path.basename(obfile)
copylist.append([os.path.join(self.task_config.comin_ges_atm, basename), obfile])

obfile = obfile.replace('satbias_cov', 'tlapse')
obfile = obfile.replace('nc4', 'txt')
basename = os.path.basename(obfile)
copylist.append([os.path.join(self.task_config.comin_ges_atm, basename), obfile])

@aerorahul aerorahul Apr 11, 2023 •
edited
Loading

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
basename = os.path.basename(obfile)
copylist.append([os.path.join(self.task_config.comin_ges_atm, basename), obfile])
obfile = obfile.replace('satbias', 'satbias_cov')
basename = os.path.basename(obfile)
copylist.append([os.path.join(self.task_config.comin_ges_atm, basename), obfile])
obfile = obfile.replace('satbias_cov', 'tlapse')
obfile = obfile.replace('nc4', 'txt')
basename = os.path.basename(obfile)
copylist.append([os.path.join(self.task_config.comin_ges_atm, basename), obfile])
obdir = os.path.dirname(obfile)
obfile = os.path.basename(obfile)
prefix = '.'.join(obfile.split('.')[:-2])
for file in ['satbias.nc4', 'satbias_cov.nc4', 'lapse.txt']:
obfile = f"{prefix}.{file}"
copylist.append([os.path.join(self.task_config.comin_ges_atm, obfile), os.path.join(obdir, obfile)])

@RussTreadon-NOAA RussTreadon-NOAA Apr 11, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@aerorahul , I like the consolidation. The suggested change, however, does not work. The bias correction files begin with the prefix gdas.t${cyc}z. For example, here's a satbias file in ROTDIR:
/scratch2/NCEPDEV/stmp1/Russ.Treadon/RUNDIRS/prufsatm/gdasatmanl_00/obs/gdas.t18z.amsua_n19.satbias.2021122118.nc4.

I can remove the 10-digit string (2021122118) from the observation yaml. We want to retain the prefix, gdas.t18z. The prefix needs to be included on the left hand side of copylist.

@CoryMartin-NOAA CoryMartin-NOAA Apr 11, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah I think I would suggest what I think you are proposing Russ:

gdas.t18z.amsua_n19.satbias.nc4
gdas.t18z.amsua_n19.satbias_cov.nc4
gdas.t18z.amsua_n19.lapse.txt

That should be fine right?

@aerorahul aerorahul Apr 11, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I agree w/ dropping the 10-character string with date. That seems superfluous.
You will need a little more even with that change. I am working on it.

@aerorahul aerorahul Apr 11, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@RussTreadon-NOAA Please try the updated suggestion.

@aerorahul aerorahul Apr 11, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I will also take a look at this yaml file and see if there is a better way to loop over these files satbias.nc4 satbias_cov.nc4 and lapse.txt.
They must be part of the bigger yaml file.

@RussTreadon-NOAA RussTreadon-NOAA Apr 11, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thank you @aerorahul . I tested a more verbose & less elegant way to form prefix. This works if I manually rename the the bias correction files in my ROTDIR. A proper test requires that I remove the GDATE string from the filename in the obs yaml and then cycle forward with this. Let me do this tomorrow plus replace my prefix with your scripting.

@RussTreadon-NOAA RussTreadon-NOAA Apr 12, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The following works

  • adapt suggested changes in analysis.py
  • remove $GDATE from bias correction filenames in observation yaml
  • rename bias correcton files in ROTDIR without $GDATE in the filename

Question: When we implement UFS-DA ATM in operations after GFS v17, what naming convention do we foresee being used in operations? Will we

  • retain $CDUMP.t${cyc}z.$file
  • include the 10-digit datetime string in the filename. For example, $CDUMP.${PDY}${cyc}.$file

Whatever naming convention operations uses, the g-w analysis.py and GDASApp observation yamls need to be consistent with one another. We have cross repo dependencies.

@RussTreadon-NOAA RussTreadon-NOAA Apr 12, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change committed at ef5da49.


bias_dict = {
'mkdir': [os.path.join(self.runtime_config['DATA'], 'bc')],

@aerorahul aerorahul Apr 11, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
'mkdir': [os.path.join(self.runtime_config['DATA'], 'bc')],
'mkdir': [os.path.join(self.runtime_config.DATA, 'bc')],

@RussTreadon-NOAA RussTreadon-NOAA Apr 12, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change committed at ef5da49.

'copy': copylist
}
return bias_dict

@logit(logger)
def add_fv3_increments(self, inc_file_tmpl: str, bkg_file_tmpl: str, incvars: List) -> None:
"""Add cubed-sphere increments to cubed-sphere backgrounds
Expand Down