Skip to content

Allow YAML input to override config.base and make HPSS_PROJECT configurable #1603

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aerorahul merged 5 commits into from
May 11, 2023
Merged

Allow YAML input to override config.base and make HPSS_PROJECT configurable #1603

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
043bd90
Have config.base updated by input YAML; make HPSS_PROJECT configurable
CoryMartin-NOAA May 10, 2023
3973209
Remove prints
CoryMartin-NOAA May 10, 2023
ce3b744
Merge branch 'NOAA-EMC:develop' into feature/base_config
CoryMartin-NOAA May 11, 2023
f9216fe
Address comments by helpful reviewer
CoryMartin-NOAA May 11, 2023
5b73269
Merge branch 'feature/base_config' of https://github.com/CoryMartin-N…
CoryMartin-NOAA May 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion parm/config/gefs/config.base.emc.dyn
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ export PARTITION_BATCH="@PARTITION_BATCH@"
export PARTITION_SERVICE="@PARTITION_SERVICE@"

# Project to use in mass store:
HPSS_PROJECT=emc-global
HPSS_PROJECT="@HPSS_PROJECT@"

# Directories relative to installation areas:
export HOMEgfs=@HOMEgfs@
Expand Down
2 changes: 1 addition & 1 deletion parm/config/gfs/config.base.emc.dyn
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ export PARTITION_BATCH="@PARTITION_BATCH@"
export PARTITION_SERVICE="@PARTITION_SERVICE@"

# Project to use in mass store:
HPSS_PROJECT=emc-global
HPSS_PROJECT="@HPSS_PROJECT@"

# Directories relative to installation areas:
export HOMEgfs=@HOMEgfs@
Expand Down
1 change: 1 addition & 0 deletions workflow/hosts/container.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ PARTITION_SERVICE: ''
CHGRP_RSTPROD: 'YES'
CHGRP_CMD: 'chgrp rstprod'
HPSSARCH: 'NO'
HPSS_PROJECT: emc-global
LOCALARCH: 'NO'
ATARDIR: '${NOSCRUB}/archive_rotdir/${PSLOT}'
MAKE_NSSTBUFR: 'NO'
Expand Down
1 change: 1 addition & 0 deletions workflow/hosts/hera.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ PARTITION_SERVICE: service
CHGRP_RSTPROD: 'YES'
CHGRP_CMD: 'chgrp rstprod'
HPSSARCH: 'YES'
HPSS_PROJECT: emc-global
LOCALARCH: 'NO'
ATARDIR: '/NCEPDEV/${HPSS_PROJECT}/1year/${USER}/${machine}/scratch/${PSLOT}'
MAKE_NSSTBUFR: 'NO'
Expand Down
1 change: 1 addition & 0 deletions workflow/hosts/jet.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ PARTITION_SERVICE: service
CHGRP_RSTPROD: 'YES'
CHGRP_CMD: 'chgrp rstprod'
HPSSARCH: 'YES'
HPSS_PROJECT: emc-global
LOCALARCH: 'NO'
ATARDIR: '/NCEPDEV/${HPSS_PROJECT}/1year/${USER}/${machine}/scratch/${PSLOT}'
MAKE_NSSTBUFR: 'NO'
Expand Down
1 change: 1 addition & 0 deletions workflow/hosts/orion.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ PARTITION_SERVICE: service
CHGRP_RSTPROD: 'YES'
CHGRP_CMD: 'chgrp rstprod'
HPSSARCH: 'NO'
HPSS_PROJECT: emc-global
LOCALARCH: 'NO'
ATARDIR: '${NOSCRUB}/archive_rotdir/${PSLOT}'
MAKE_NSSTBUFR: 'NO'
Expand Down
1 change: 1 addition & 0 deletions workflow/hosts/s4.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ PARTITION_SERVICE: serial
CHGRP_RSTPROD: 'NO'
CHGRP_CMD: 'ls'
HPSSARCH: 'NO'
HPSS_PROJECT: emc-global
LOCALARCH: 'NO'
ATARDIR: '${NOSCRUB}/archive_rotdir/${PSLOT}'
MAKE_NSSTBUFR: 'YES'
Expand Down
1 change: 1 addition & 0 deletions workflow/hosts/wcoss2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ PARTITION_SERVICE: ''
CHGRP_RSTPROD: 'YES'
CHGRP_CMD: 'chgrp rstprod'
HPSSARCH: 'NO'
HPSS_PROJECT: emc-global
LOCALARCH: 'NO'
ATARDIR: '/NCEPDEV/${HPSS_PROJECT}/1year/${USER}/${machine}/scratch/${PSLOT}'
MAKE_NSSTBUFR: 'NO'
Expand Down
16 changes: 12 additions & 4 deletions workflow/setup_expt.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -227,22 +227,25 @@ def fill_EXPDIR(inputs):

def update_configs(host, inputs):

# First update config.base
edit_baseconfig(host, inputs)

# Read in the YAML file to fill out templates and override host defaults
yaml_path = inputs.yaml
yaml_dict = YAMLFile(path=yaml_path)

# First update config.base
edit_baseconfig(host, inputs, yaml_dict)

# loop over other configs and update them
for cfg in yaml_dict.keys():
if cfg == 'base':
continue
cfg_file = f'{inputs.expdir}/{inputs.pslot}/config.{cfg}'
cfg_dict = get_template_dict(yaml_dict[cfg])
edit_config(cfg_file, cfg_file, cfg_dict)

return


def edit_baseconfig(host, inputs):
def edit_baseconfig(host, inputs, yaml_dict):
"""
Parses and populates the templated `config.base.emc.dyn` to `config.base`
"""
Expand Down Expand Up @@ -289,6 +292,11 @@ def edit_baseconfig(host, inputs):
extend_dict = {"@CCPP_SUITE@": "FV3_GFS_v17_p8", "@IMP_PHYSICS@": 8}
tmpl_dict = dict(tmpl_dict, **extend_dict)

try:
tmpl_dict = dict(tmpl_dict, **get_template_dict(yaml_dict['base']))
except KeyError:
pass

base_input = f'{inputs.configdir}/config.base.emc.dyn'
base_output = f'{inputs.expdir}/{inputs.pslot}/config.base'
edit_config(base_input, base_output, tmpl_dict)
Expand Down