Add capability to create aerosol AOD analysis products in GRIB2 format via UPP#3889
Add capability to create aerosol AOD analysis products in GRIB2 format via UPP#3889ypwang19 wants to merge 57 commits into
Conversation
…tprocess * upstream/develop: Refactor marine ensemble recentering script and move to GW from GDASApp (NOAA-EMC#3882)
|
I believe @bbakernoaa said the UPP hash has been updated in the UFS-Weather-Model now. I also see several open PRs in the global-workflow that are UPP related, could these be combined? |
Thanks for reminding me about the UPP hash, I just updated that. |
ypwang19
requested review from
CoryMartin-NOAA,
DavidHuber-NOAA,
DavidNew-NOAA and
aerorahul
as code owners
DavidNew-NOAA
left a comment
DavidNew-NOAA
left a comment
There was a problem hiding this comment.
A few comments. Making UPP_CONFIG_YAML a key inside of TASK_CONFIG_YAML would require changes inside of NOAA-EMC/GDASApp#1964, but wait for @aerorahul's feedback about invoking the upp object in exglobal_aero_analysis_execute.py before making those changes, since in that case, UPP_CONFIG_YAML should indeed be a separate yaml.
| AeroAnl = AerosolAnalysis(config) | ||
|
|
||
| # Process aerosal products through UPP | ||
| AeroAnl.upp_anlproc() |
There was a problem hiding this comment.
Finalize jobs generally involve moving files to COM or archiving/compressing files. This method involves computing analyses and processing those analyses. I think this should be moved to the execute job (exglobal_aero_analysis_execute.py)
| upp_yaml = upp.task_config.upp_yaml | ||
| upp.initialize(upp_yaml) | ||
|
|
||
| upp_dict = AttrDict() |
There was a problem hiding this comment.
There probably isn't a reason to create a separate dict for UPP as long as there are no conflicts. You can just pass task_config to it, making sure task_config has everything the upp object needs.
|
|
||
| # ---- add atmo increments to atmf000 files | ||
| logger.info('Adding atmo increments to RESTART files') | ||
| inc_file = os.path.join(upp_dict.DATA, f"{self.task_config.APREFIX}increment.atm.i006.nc") |
There was a problem hiding this comment.
Best to put filenames like this in task_config in the constructor
| # ---- add aero increments to atm analysis files | ||
| logger.info('Adding aero increments to RESTART files') | ||
| bkg_file = os.path.join(upp_dict.DATA, f"{self.task_config.GPREFIX }atm.f006.nc") | ||
| inc_filename = f"aeroinc_gauss.{self.task_config.current_cycle.strftime('%Y-%m-%dT%H:%M:%S')}Z.gaussian.modelLevels.nc" |
There was a problem hiding this comment.
Use to_isotime from wxflow here
| with Dataset(file, mode='a') as rstfile: | ||
| time = rstfile.variables['time'] | ||
| time[:] = 0.0 | ||
| time.setncattr("units", f"hours since {self.task_config.current_cycle.strftime('%Y-%m-%d %H:%M:%S')}") |
There was a problem hiding this comment.
If it's ok for the format to be slightly different for this attribute (no space in the middle and a "Z" at the end), you can just use to_isotime from wxflow
| increment = incfile.variables[incname][:] | ||
| # handel latitude inversion: atminc: lat=-90 to +90, atmbkg: lat=+90 to -90 | ||
| lat_axis_index = 1 | ||
| increment_lat_inversion = np.flip(increment, axis=lat_axis_index) |
There was a problem hiding this comment.
Double check that you still need to do this since @RussTreadon-NOAA fixed the inverted axes in https://github.com/JCSDA-internal/fv3-jedi/pull/1428
| export TASK_CONFIG_YAML="${PARMgfs}/gdas/aero/aero_det_config.yaml.j2" | ||
| export OBS_LIST_YAML="${PARMgfs}/gdas/aero/aero_obs_list.yaml.j2" | ||
| export BIAS_FILES_YAML="${PARMgfs}/gdas/aero/aero_bias_files.yaml.j2" | ||
| export UPP_CONFIG_YAML="${PARMgfs}/gdas/aero/aero_upp_config.yaml.j2" |
There was a problem hiding this comment.
If this is all part of the same task, then this whole yaml should just me a key in TASK_CONFIG_YAML (aero_det_config.yaml.j2). That's the reason for #4080, to have a single configuration yaml for each task
| - Execute upp.x | ||
| """ | ||
|
|
||
| upp = UPP(self.task_config) |
There was a problem hiding this comment.
I'm torn about invoking a Task object (upp) inside of another Task object (AerosolAnalysis inherits from Analysis which inherits from Task). I'm wondering if this should be invoked inside the exglobal_aero_analysis_execute.py after running AerosolAnalysis.execute(). That might be awkward though, because it would require quite a bit of code to live in exglobal_aero_analysis_execute.py in order to initialize the upp object. I would like to know what @aerorahul thinks
| AeroAnl = AerosolAnalysis(config) | ||
|
|
||
| # Process aerosal products through UPP | ||
| AeroAnl.upp_anlproc() |
There was a problem hiding this comment.
The UPP should be a different job and task in the workflow and not part of the aerosol analysis job. The aerosol analysis task is the dependency of the aerosol UPP
There was a problem hiding this comment.
Yes, good point. I don't know how long UPP takes to run, but if it does take a while, then putting it in the aerosol analysis task will delay the forecast job from running
5 participants
Description
This PR resolves issue #3888
This PR adds the capability to process analysis tracer fields into aerosol AOD product in GRIB2 format using the
upptool within the GCAFS workflow. Aerosol variables in atmospheric analysis file (analysis.atm.a006.nc) are all missing data and not updated, such that the analysis fields on the Gaussian grid are generated by adding aerosol increments (aeroinc_gaussian) and atmospheric increments (increment.atm.i006.nc) to theatm.f006background file.This feature is integrated into the
aeroanlfinaljob.A new GCAFS control file has been updated in UPP. The UPP hash has been updated in ufs model and fv3atm.
This PR requires updates in GDASApp NOAA-EMC/GDASApp#1964
Type of change
Change characteristics
How has this been tested?
Checklist