Skip to content

Add JEDI atmosphere only CI #2357

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
WalterKolczynski-NOAA merged 33 commits into from
Mar 12, 2024
Merged

Add JEDI atmosphere only CI #2357

Show file tree
Hide file tree
Changes from 30 commits
Commits
Show all changes
33 commits
Select commit Hold shift + click to select a range
4d8b4ee
minimal set of changes to execute JEDI ATM only CI case (#2294)
RussTreadon-NOAA Feb 26, 2024
7a7b2ac
Merge branch 'NOAA-EMC:develop' into feature/ufsda_atm_ci
RussTreadon-NOAA Feb 26, 2024
90e605e
Merge branch 'NOAA-EMC:develop' into feature/ufsda_atm_ci
RussTreadon-NOAA Feb 27, 2024
907e475
Merge branch 'NOAA-EMC:develop' into feature/ufsda_atm_ci
RussTreadon-NOAA Feb 28, 2024
dcd79ee
Merge branch 'NOAA-EMC:develop' into feature/ufsda_atm_ci
RussTreadon-NOAA Feb 29, 2024
a84a484
Merge branch 'NOAA-EMC:develop' into feature/ufsda_atm_ci
RussTreadon-NOAA Feb 29, 2024
b13a8d4
replace nc4 with nc in setup_exp.py, correct JEDI DA var and ens bug …
RussTreadon-NOAA Feb 29, 2024
16199a0
Merge branch 'NOAA-EMC:develop' into feature/ufsda_atm_ci
RussTreadon-NOAA Mar 1, 2024
5a1db99
adjust JEDI atm var configuration (#2294)
RussTreadon-NOAA Mar 1, 2024
46009ec
Merge branch 'develop' into feature/ufsda_atm_ci
RussTreadon-NOAA Mar 1, 2024
be45b07
Merge branch 'develop' into feature/ufsda_atm_ci
RussTreadon-NOAA Mar 1, 2024
1b44f6e
remove atms_n20 from setup_expt.py (#2294)
RussTreadon-NOAA Mar 2, 2024
811620f
Merge branch 'develop' into feature/ufsda_atm_ci
RussTreadon-NOAA Mar 2, 2024
e4d2eea
Merge branch 'develop' into feature/ufsda_atm_ci
RussTreadon-NOAA Mar 5, 2024
db669cf
set JEDI ATM CASE_ANL based on DOHYBVAR, replace C48C48 JEDI ATM case…
RussTreadon-NOAA Mar 5, 2024
0aeca40
address shellcheck error (#2294)
RussTreadon-NOAA Mar 5, 2024
b59662e
properly indent shellcheck directive (#2294)
RussTreadon-NOAA Mar 5, 2024
1b289bf
clean up comments in JEDI ATM Var and Ens init and run config files (…
RussTreadon-NOAA Mar 5, 2024
69ade61
Merge branch 'NOAA-EMC:develop' into feature/ufsda_atm_ci
RussTreadon-NOAA Mar 5, 2024
99b21d4
update JEDI ATM templates to jinja2 (#2294)
RussTreadon-NOAA Mar 6, 2024
04bbdc6
more jinja updates for JEDI ATM, increase JEDI ATMENS layout for CI t…
RussTreadon-NOAA Mar 6, 2024
fd2e780
Merge branch 'develop' into feature/ufsda_atm_ci
RussTreadon-NOAA Mar 7, 2024
ae103a2
Merge branch 'develop' into feature/ufsda_atm_ci
RussTreadon-NOAA Mar 7, 2024
4f4fdd9
correct syntax errors in atm_jedi_fix.yaml (#2294)
RussTreadon-NOAA Mar 7, 2024
e1ab5fe
Merge branch 'develop' into feature/ufsda_atm_ci
RussTreadon-NOAA Mar 8, 2024
80639a0
turn off JEDI ATM CI on hera and orion (#2294)
RussTreadon-NOAA Mar 8, 2024
ed04239
Replace ATMENSYAML and ATMVARYAML with JEDIYAML (#2294)
RussTreadon-NOAA Mar 8, 2024
c571fb6
Merge branch 'develop' into feature/ufsda_atm_ci
RussTreadon-NOAA Mar 8, 2024
dd44e30
add JEDI ATM CI, update JEDI ATM to jinja2 (#2294)
RussTreadon-NOAA Mar 11, 2024
563a438
Source required JEDI ATM configs in applications.py (#2294)
RussTreadon-NOAA Mar 11, 2024
193be93
streamline sourcing of JEDI ATM configs in applications.py (#2294)
RussTreadon-NOAA Mar 11, 2024
36289ee
replace HOMEgfs in parm path with PARMgfs (#2294)
RussTreadon-NOAA Mar 11, 2024
de6e459
Merge branch 'develop' into feature/ufsda_atm_ci
RussTreadon-NOAA Mar 11, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions ci/cases/pr/C48C48_ufs_hybatmDA.yaml → ci/cases/pr/C96C48_ufs_hybatmDA.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,13 @@ experiment:
arguments:
pslot: {{ 'pslot' | getenv }}
app: ATM
resdetatmos: 48
resdetatmos: 96
resensatmos: 48
comroot: {{ 'RUNTESTS' | getenv }}/COMROOT
expdir: {{ 'RUNTESTS' | getenv }}/EXPDIR
idate: 2021032312
edate: 2021032400
icsdir: {{ 'ICSDIR_ROOT' | getenv }}/C96C48
idate: 2024022318
edate: 2024022400
nens: 2
gfs_cyc: 1
start: warm
Expand Down
8 changes: 4 additions & 4 deletions ci/cases/yamls/ufs_hybatmDA_defaults.ci.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6,11 +6,11 @@ base:
DO_JEDIATMENS: "YES"
ACCOUNT: {{ 'SLURM_ACCOUNT' | getenv }}
atmanl:
LAYOUT_X_ATMANL: 1
LAYOUT_Y_ATMANL: 1
LAYOUT_X_ATMANL: 4
LAYOUT_Y_ATMANL: 4
atmensanl:
LAYOUT_X_ATMENSANL: 1
LAYOUT_Y_ATMENSANL: 1
LAYOUT_X_ATMENSANL: 4
LAYOUT_Y_ATMENSANL: 4
esfc:
DONST: "NO"
nsst:
Expand Down
14 changes: 9 additions & 5 deletions parm/config/gfs/config.atmanl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,12 +5,16 @@

echo "BEGIN: config.atmanl"

export CASE_ANL=${CASE}
export OBS_YAML_DIR=${HOMEgfs}/sorc/gdas.cd/parm/atm/obs/config/
export OBS_LIST=${HOMEgfs}/sorc/gdas.cd/parm/atm/obs/lists/gdas_prototype_3d.yaml
export ATMVARYAML=${HOMEgfs}/sorc/gdas.cd/parm/atm/variational/3dvar_drpcg.yaml
if [[ ${DOHYBVAR} = "YES" ]]; then
# shellcheck disable=SC2153
export CASE_ANL=${CASE_ENS}
Comment thread Fixed
else
export CASE_ANL=${CASE}
fi
export OBS_LIST="${HOMEgfs}/parm/gdas/atm/obs/lists/gdas_prototype_3d.yaml.j2"
Comment thread
RussTreadon-NOAA marked this conversation as resolved.
Outdated
export JEDIYAML="${HOMEgfs}/parm/gdas/atm/variational/3dvar_drpcg.yaml.j2"
Comment thread
RussTreadon-NOAA marked this conversation as resolved.
Outdated
export STATICB_TYPE="gsibec"
export BERROR_YAML=${HOMEgfs}/sorc/gdas.cd/parm/atm/berror/staticb_${STATICB_TYPE}.yaml
export BERROR_YAML="${HOMEgfs}/parm/gdas/atm/berror/hybvar_${STATICB_TYPE}.yaml.j2"
Comment thread
RussTreadon-NOAA marked this conversation as resolved.
Outdated
export INTERP_METHOD='barycentric'

export layout_x_atmanl=@LAYOUT_X_ATMANL@
Expand Down
1 change: 1 addition & 0 deletions parm/config/gfs/config.atmanlinit
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@ echo "BEGIN: config.atmanlinit"

# Get task specific resources
. "${EXPDIR}/config.resources" atmanlinit
Comment thread
RussTreadon-NOAA marked this conversation as resolved.

echo "END: config.atmanlinit"
5 changes: 2 additions & 3 deletions parm/config/gfs/config.atmensanl
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -5,9 +5,8 @@

echo "BEGIN: config.atmensanl"

export OBS_YAML_DIR=${HOMEgfs}/sorc/gdas.cd/parm/atm/obs/config/
export OBS_LIST=${HOMEgfs}/sorc/gdas.cd/parm/atm/obs/lists/lgetkf_prototype.yaml
export ATMENSYAML=${HOMEgfs}/sorc/gdas.cd/parm/atm/lgetkf/lgetkf.yaml
export OBS_LIST="${HOMEgfs}/parm/gdas/atm/obs/lists/lgetkf_prototype.yaml.j2"
Comment thread
RussTreadon-NOAA marked this conversation as resolved.
Outdated
export JEDIYAML="${HOMEgfs}/parm/gdas/atm/lgetkf/lgetkf.yaml.j2"
Comment thread
RussTreadon-NOAA marked this conversation as resolved.
Outdated
export INTERP_METHOD='barycentric'

export layout_x_atmensanl=@LAYOUT_X_ATMENSANL@
Expand Down
1 change: 1 addition & 0 deletions parm/config/gfs/config.atmensanlinit
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -7,4 +7,5 @@ echo "BEGIN: config.atmensanlinit"

# Get task specific resources
. "${EXPDIR}/config.resources" atmensanlinit

echo "END: config.atmensanlinit"
10 changes: 5 additions & 5 deletions parm/gdas/atm_jedi_fix.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
mkdir:
- $(DATA)/fv3jedi
- '{{ DATA }}/fv3jedi'
copy:
- [$(FIXgfs)/gdas/fv3jedi/fv3files/akbk$(npz).nc4, $(DATA)/fv3jedi/akbk.nc4]
- [$(FIXgfs)/gdas/fv3jedi/fv3files/fmsmpp.nml, $(DATA)/fv3jedi/fmsmpp.nml]
- [$(FIXgfs)/gdas/fv3jedi/fv3files/field_table_gfdl, $(DATA)/fv3jedi/field_table]
- [$(HOMEgfs)/sorc/gdas.cd/parm/io/fv3jedi_fieldmetadata_restart.yaml, $(DATA)/fv3jedi/fv3jedi_fieldmetadata_restart.yaml]
- ['{{ FIXgfs }}/gdas/fv3jedi/fv3files/akbk{{ npz }}.nc4', '{{ DATA }}/fv3jedi/akbk.nc4']
- ['{{ FIXgfs }}/gdas/fv3jedi/fv3files/fmsmpp.nml', '{{ DATA }}/fv3jedi/fmsmpp.nml']
- ['{{ FIXgfs }}/gdas/fv3jedi/fv3files/field_table_gfdl', '{{ DATA }}/fv3jedi/field_table']
- ['{{ HOMEgfs }}/sorc/gdas.cd/parm/io/fv3jedi_fieldmetadata_restart.yaml', '{{ DATA }}/fv3jedi/fv3jedi_fieldmetadata_restart.yaml']
Comment thread
RussTreadon-NOAA marked this conversation as resolved.
Outdated
2 changes: 1 addition & 1 deletion sorc/gdas.cd
Open in desktop
Submodule gdas.cd updated 122 files
4 changes: 2 additions & 2 deletions ush/python/pygfs/task/atm_analysis.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
FileHandler,
add_to_datetime, to_fv3time, to_timedelta, to_YMDH,
chdir,
parse_yamltmpl, parse_j2yaml, save_as_yaml,
parse_j2yaml, save_as_yaml,
logit,
Executable,
WorkflowException)
Expand Down Expand Up @@ -103,7 +103,7 @@ def initialize(self: Analysis) -> None:

# generate variational YAML file
logger.debug(f"Generate variational YAML file: {self.task_config.fv3jedi_yaml}")
varda_yaml = parse_j2yaml(self.task_config.ATMVARYAML, self.task_config)
varda_yaml = parse_j2yaml(self.task_config.JEDIYAML, self.task_config, searchpath=self.gdasapp_j2tmpl_dir)
save_as_yaml(varda_yaml, self.task_config.fv3jedi_yaml)
logger.info(f"Wrote variational YAML to: {self.task_config.fv3jedi_yaml}")

Expand Down
4 changes: 2 additions & 2 deletions ush/python/pygfs/task/atmens_analysis.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,7 @@
FileHandler,
add_to_datetime, to_fv3time, to_timedelta, to_YMDH, to_YMD,
chdir,
parse_yamltmpl, parse_j2yaml, save_as_yaml,
parse_j2yaml, save_as_yaml,
logit,
Executable,
WorkflowException,
Expand Down Expand Up @@ -119,7 +119,7 @@ def initialize(self: Analysis) -> None:

# generate ensemble da YAML file
logger.debug(f"Generate ensemble da YAML file: {self.task_config.fv3jedi_yaml}")
ensda_yaml = parse_j2yaml(self.task_config.ATMENSYAML, self.task_config)
ensda_yaml = parse_j2yaml(self.task_config.JEDIYAML, self.task_config, searchpath=self.gdasapp_j2tmpl_dir)
save_as_yaml(ensda_yaml, self.task_config.fv3jedi_yaml)
logger.info(f"Wrote ensemble da YAML to: {self.task_config.fv3jedi_yaml}")

Expand Down
8 changes: 8 additions & 0 deletions workflow/applications/applications.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -145,6 +145,14 @@ def _source_configs(self, conf: Configuration) -> Dict[str, Any]:
files += ['config.anal', 'config.eupd']
elif config in ['efcs']:
files += ['config.fcst', 'config.efcs']
elif config in ['atmanlinit']:
files += ['config.atmanl', 'config.atmanlinit']
elif config in ['atmanlrun']:
files += ['config.atmanl', 'config.atmanlrun']
elif config in ['atmensanlinit']:
files += ['config.atmensanl', 'config.atmensanlinit']
elif config in ['atmensanlrun']:
files += ['config.atmensanl', 'config.atmensanlrun']
Comment thread
RussTreadon-NOAA marked this conversation as resolved.
Outdated
elif 'wave' in config:
files += ['config.wave', f'config.{config}']
else:
Expand Down