Skip to content

Update UFSDA ATM ens for new COM directory structure #1538

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
WalterKolczynski-NOAA merged 13 commits into from
Apr 28, 2023
Merged

Update UFSDA ATM ens for new COM directory structure #1538

Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
87630e7
initial refactoring of UFSDA ATM ensemble analysis for COM refactor (…
RussTreadon-NOAA Apr 26, 2023
60b4602
fix linter error and remove exports in UFSDA ATM ens jobs (#1518)
RussTreadon-NOAA Apr 26, 2023
0f1c2cf
Merge branch 'NOAA-EMC:develop' into feature/ufsda_atmens_com
RussTreadon-NOAA Apr 27, 2023
3253bfd
use Template to build restart paths in atmens_analysis (#1518)
RussTreadon-NOAA Apr 27, 2023
dc0f196
remove COM_TOP references, use Template to build paths in atmens_anal…
RussTreadon-NOAA Apr 27, 2023
52bd60b
Merge branch 'NOAA-EMC:develop' into feature/ufsda_atmens_com
RussTreadon-NOAA Apr 27, 2023
383877e
Merge branch 'NOAA-EMC:develop' into feature/ufsda_atmens_com
RussTreadon-NOAA Apr 27, 2023
da20985
add get_bkg_ens_dict to analysis.py, remove get_bkg_dict from atmens_…
RussTreadon-NOAA Apr 28, 2023
37fbac4
move template dictionary definition outside ensemble member loop (#1518)
RussTreadon-NOAA Apr 28, 2023
8126791
refactor ensemble get_bkg_dict (#1518)
RussTreadon-NOAA Apr 28, 2023
292acce
move creation of member analysis output directories to initialize (#1…
RussTreadon-NOAA Apr 28, 2023
4402e25
replace self.runtime_config with self.task_config (#1518)
RussTreadon-NOAA Apr 28, 2023
36d733d
remove unused modules.
aerorahul Apr 28, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
12 changes: 5 additions & 7 deletions jobs/JGLOBAL_ATMENS_ANALYSIS_FINALIZE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -8,19 +8,17 @@ source "${HOMEgfs}/ush/jjob_header.sh" -e "atmensanlfinal" -c "base atmensanl at
##############################################
# Set variables used in the script
##############################################
GDATE=$(date +%Y%m%d%H -d "${PDY} ${cyc} - ${assim_freq} hours")
GDUMP="gdas"
GDUMP_ENS="enkf${GDUMP}"

##############################################
# Begin JOB SPECIFIC work
##############################################
# Generate COM variable from template
MEMDIR='ensstat' RUN=${GDUMP_ENS} YMD=${PDY} HH=${cyc} generate_com -rx \
COM_ATMOS_ANALYSIS_ENS:COM_ATMOS_ANALYSIS_TMPL

export COMOUT=${COMOUT:-${ROTDIR}/${RUN}.${PDY}/${cyc}}
mkdir -p "${COMOUT}"

# COMIN_GES and COMIN_GES_ENS are used in script
export COMIN_GES="${ROTDIR}/${GDUMP}.${GDATE:0:8}/${GDATE:8:2}/atmos"
export COMIN_GES_ENS="${ROTDIR}/enkf${GDUMP}.${GDATE:0:8}/${GDATE:8:2}"
mkdir -m 755 -p "${COM_ATMOS_ANALYSIS_ENS}"

###############################################################
# Run relevant script
Expand Down
13 changes: 7 additions & 6 deletions jobs/JGLOBAL_ATMENS_ANALYSIS_INITIALIZE
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,19 +7,20 @@ source "${HOMEgfs}/ush/jjob_header.sh" -e "atmensanlinit" -c "base atmensanl atm
##############################################
# Set variables used in the script
##############################################
# shellcheck disable=SC2153
GDATE=$(date +%Y%m%d%H -d "${PDY} ${cyc} - ${assim_freq} hours")
gPDY=${GDATE:0:8}
gcyc=${GDATE:8:2}
GDUMP="gdas"


##############################################
# Begin JOB SPECIFIC work
##############################################
export COMOUT=${COMOUT:-${ROTDIR}/${RUN}.${PDY}/${cyc}}
mkdir -p "${COMOUT}"
# Generate COM variables from templates
RUN=${GDUMP} YMD=${PDY} HH=${cyc} generate_com -rx COM_OBS

# COMIN_GES and COMIN_GES_ENS are used in script
export COMIN_GES="${ROTDIR}/${GDUMP}.${GDATE:0:8}/${GDATE:8:2}/atmos"
export COMIN_GES_ENS="${ROTDIR}/enkf${GDUMP}.${GDATE:0:8}/${GDATE:8:2}"
RUN=${GDUMP} YMD=${gPDY} HH=${gcyc} generate_com -rx \
COM_ATMOS_ANALYSIS_PREV:COM_ATMOS_ANALYSIS_TMPL

###############################################################
# Run relevant script
Expand Down
3 changes: 0 additions & 3 deletions jobs/JGLOBAL_ATMENS_ANALYSIS_RUN
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -13,9 +13,6 @@ source "${HOMEgfs}/ush/jjob_header.sh" -e "atmensanlrun" -c "base atmensanl atme
# Begin JOB SPECIFIC work
##############################################

export COMOUT=${COMOUT:-${ROTDIR}/${RUN}.${PDY}/${cyc}}
mkdir -p "${COMOUT}"

###############################################################
# Run relevant script

Expand Down
62 changes: 43 additions & 19 deletions ush/python/pygfs/task/atmens_analysis.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9,12 +9,13 @@

from pygw.attrdict import AttrDict
from pygw.file_utils import FileHandler
from pygw.timetools import add_to_datetime, to_fv3time, to_timedelta, to_YMDH
from pygw.timetools import add_to_datetime, to_fv3time, to_timedelta, to_YMDH, to_YMD
from pygw.fsutils import rm_p, chdir
from pygw.yaml_file import parse_yamltmpl, parse_j2yaml, save_as_yaml
from pygw.logger import logit
from pygw.executable import Executable
from pygw.exceptions import WorkflowException
from pygw.template import Template, TemplateConstants
from pygfs.task.analysis import Analysis

logger = getLogger(__name__.split('.')[-1])
Expand Down Expand Up @@ -45,8 +46,6 @@ def __init__(self, config):
'npz_anl': self.config.LEVS - 1,
'ATM_WINDOW_BEGIN': _window_begin,
'ATM_WINDOW_LENGTH': f"PT{self.config.assim_freq}H",
'comin_ges_atm': self.config.COMIN_GES,
'comin_ges_atmens': self.config.COMIN_GES_ENS,
'OPREFIX': f"{self.config.EUPD_CYC}.t{self.runtime_config.cyc:02d}z.", # TODO: CDUMP is being replaced by RUN
'APREFIX': f"{self.runtime_config.CDUMP}.t{self.runtime_config.cyc:02d}z.", # TODO: CDUMP is being replaced by RUN
'GPREFIX': f"gdas.t{self.runtime_config.previous_cycle.hour:02d}z.",
Expand Down Expand Up @@ -169,7 +168,7 @@ def finalize(self: Analysis) -> None:
"""
# ---- tar up diags
# path of output tar statfile
atmensstat = os.path.join(self.task_config.COMOUT, f"{self.task_config.APREFIX}atmensstat")
atmensstat = os.path.join(self.task_config.COM_ATMOS_ANALYSIS_ENS, f"{self.task_config.APREFIX}atmensstat")

# get list of diag files to put in tarball
diags = glob.glob(os.path.join(self.task_config.DATA, 'diags', 'diag*nc4'))
Expand All @@ -190,12 +189,12 @@ def finalize(self: Analysis) -> None:
archive.add(diaggzip, arcname=os.path.basename(diaggzip))

# copy full YAML from executable to ROTDIR
logger.info(f"Copying {self.task_config.fv3jedi_yaml} to {self.task_config.COMOUT}")
logger.info(f"Copying {self.task_config.fv3jedi_yaml} to {self.task_config.COM_ATMOS_ANALYSIS_ENS}")
src = os.path.join(self.task_config.DATA, f"{self.task_config.CDUMP}.t{self.task_config.cyc:02d}z.atmens.yaml")
dest = os.path.join(self.task_config.COMOUT, f"{self.task_config.CDUMP}.t{self.task_config.cyc:02d}z.atmens.yaml")
dest = os.path.join(self.task_config.COM_ATMOS_ANALYSIS_ENS, f"{self.task_config.CDUMP}.t{self.task_config.cyc:02d}z.atmens.yaml")
logger.debug(f"Copying {src} to {dest}")
yaml_copy = {
'mkdir': [self.task_config.COMOUT],
'mkdir': [self.task_config.COM_ATMOS_ANALYSIS_ENS],
'copy': [[src, dest]]
}
FileHandler(yaml_copy).sync()
Expand All @@ -211,7 +210,7 @@ def clean(self):
def get_bkg_dict(self, task_config: Dict[str, Any]) -> Dict[str, List[str]]:
"""Compile a dictionary of model background files to copy

This method constructs a dictionary of FV3 RESTART files (coupler, core, tracer)
This method constructs a dictionary of FV3 restart files (coupler, core, tracer)
that are needed for global atmens DA and returns said dictionary for use by the FileHandler class.

Parameters
Expand All @@ -224,7 +223,7 @@ def get_bkg_dict(self, task_config: Dict[str, Any]) -> Dict[str, List[str]]:
bkg_dict: Dict
a dictionary containing the list of model background files to copy for FileHandler
"""
# NOTE for now this is FV3 RESTART files and just assumed to be fh006
# NOTE for now this is FV3 restart files and just assumed to be fh006
# loop over ensemble members
dirlist = []
bkglist = []
Expand All @@ -234,8 +233,20 @@ def get_bkg_dict(self, task_config: Dict[str, Any]) -> Dict[str, List[str]]:
# accumulate directory list for member restart files
dirlist.append(os.path.join(task_config.DATA, 'bkg', memchar))

# get FV3 RESTART files, this will be a lot simpler when using history files
rst_dir = os.path.join(task_config.comin_ges_atmens, memchar, 'atmos/RESTART')
# get FV3 restart files, this will be a lot simpler when using history files
template = task_config.COM_ATMOS_RESTART_TMPL
tmpl_dict = {
'ROTDIR': task_config.ROTDIR,
'RUN': self.runtime_config.RUN,
'YMD': to_YMD(task_config.previous_cycle),
'HH': task_config.previous_cycle.strftime('%H')
}
Comment thread
RussTreadon-NOAA marked this conversation as resolved.
Outdated

# get FV3 restart files, this will be a lot simpler when using history files
tmpl_dict['MEMDIR'] = memchar
rst_dir = Template.substitute_structure(template, TemplateConstants.DOLLAR_CURLY_BRACE, tmpl_dict.get)
os.makedirs(rst_dir, mode=0o775, exist_ok=True)
Comment thread
RussTreadon-NOAA marked this conversation as resolved.
Outdated

run_dir = os.path.join(task_config.DATA, 'bkg', memchar)

# atmens DA needs coupler
Expand Down Expand Up @@ -289,17 +300,30 @@ def jedi2fv3inc(self: Analysis) -> None:
memchar = f"mem{imem:03d}"

# make output directory for member increment
incdir = [
os.path.join(self.task_config.COMOUT, memchar, 'atmos')
]
FileHandler({'mkdir': incdir}).sync()
template = self.task_config.COM_ATMOS_ANALYSIS_TMPL
tmpl_dict = {
Comment thread
RussTreadon-NOAA marked this conversation as resolved.
Outdated
'ROTDIR': self.task_config.ROTDIR,
'RUN': self.runtime_config.RUN,
'YMD': to_YMD(self.task_config.current_cycle),
'HH': self.task_config.current_cycle.strftime('%H')
}
tmpl_dict['MEMDIR'] = memchar
incdir = Template.substitute_structure(template, TemplateConstants.DOLLAR_CURLY_BRACE, tmpl_dict.get)
FileHandler({'mkdir': [incdir]}).sync()
Comment thread
RussTreadon-NOAA marked this conversation as resolved.
Outdated

# rewrite UFS-DA atmens increments
atmges_fv3 = os.path.join(self.task_config.COMIN_GES_ENS, memchar, 'atmos',
f"{self.task_config.CDUMP}.t{self.runtime_config.previous_cycle.hour:02d}z.atmf006.nc")
template = self.task_config.COM_ATMOS_HISTORY_TMPL
tmpl_dict = {
'ROTDIR': self.task_config.ROTDIR,
'RUN': self.runtime_config.RUN,
'YMD': to_YMD(self.task_config.previous_cycle),
'HH': self.task_config.previous_cycle.strftime('%H')
}
tmpl_dict['MEMDIR'] = memchar
gesdir = Template.substitute_structure(template, TemplateConstants.DOLLAR_CURLY_BRACE, tmpl_dict.get)
atmges_fv3 = os.path.join(gesdir, f"{self.task_config.CDUMP}.t{self.runtime_config.previous_cycle.hour:02d}z.atmf006.nc")
Comment thread
RussTreadon-NOAA marked this conversation as resolved.
Outdated
atminc_jedi = os.path.join(self.task_config.DATA, 'anl', memchar, f'atminc.{cdate_inc}z.nc4')
atminc_fv3 = os.path.join(self.task_config.COMOUT, memchar, 'atmos',
f"{self.task_config.CDUMP}.t{self.runtime_config.cyc:02d}z.atminc.nc")
atminc_fv3 = os.path.join(incdir, f"{self.task_config.CDUMP}.t{self.runtime_config.cyc:02d}z.atminc.nc")
Comment thread
RussTreadon-NOAA marked this conversation as resolved.
Outdated

# Execute incpy to create the UFS model atm increment file
# TODO: use MPMD or parallelize with mpi4py
Expand Down