Skip to content

Allow YAML input to override config.base and make HPSS_PROJECT configurable #1603

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
aerorahul merged 5 commits into from
May 11, 2023
Merged

Allow YAML input to override config.base and make HPSS_PROJECT configurable #1603

Show file tree
Hide file tree
Changes from 3 commits
Commits
Show all changes
5 commits
Select commit Hold shift + click to select a range
043bd90
Have config.base updated by input YAML; make HPSS_PROJECT configurable
CoryMartin-NOAA May 10, 2023
3973209
Remove prints
CoryMartin-NOAA May 10, 2023
ce3b744
Merge branch 'NOAA-EMC:develop' into feature/base_config
CoryMartin-NOAA May 11, 2023
f9216fe
Address comments by helpful reviewer
CoryMartin-NOAA May 11, 2023
5b73269
Merge branch 'feature/base_config' of https://github.com/CoryMartin-N…
CoryMartin-NOAA May 11, 2023
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion parm/config/gefs/config.base.emc.dyn
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ export PARTITION_BATCH="@PARTITION_BATCH@"
export PARTITION_SERVICE="@PARTITION_SERVICE@"

# Project to use in mass store:
HPSS_PROJECT=emc-global
HPSS_PROJECT="@HPSS_PROJECT@"

# Directories relative to installation areas:
export HOMEgfs=@HOMEgfs@
Expand Down
2 changes: 1 addition & 1 deletion parm/config/gfs/config.base.emc.dyn
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ export PARTITION_BATCH="@PARTITION_BATCH@"
export PARTITION_SERVICE="@PARTITION_SERVICE@"

# Project to use in mass store:
HPSS_PROJECT=emc-global
HPSS_PROJECT="@HPSS_PROJECT@"

# Directories relative to installation areas:
export HOMEgfs=@HOMEgfs@
Expand Down
1 change: 1 addition & 0 deletions workflow/hosts/container.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ PARTITION_SERVICE: ''
CHGRP_RSTPROD: 'YES'
CHGRP_CMD: 'chgrp rstprod'
HPSSARCH: 'NO'
HPSS_PROJECT: emc-global
LOCALARCH: 'NO'
ATARDIR: '${NOSCRUB}/archive_rotdir/${PSLOT}'
MAKE_NSSTBUFR: 'NO'
Expand Down
1 change: 1 addition & 0 deletions workflow/hosts/hera.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ PARTITION_SERVICE: service
CHGRP_RSTPROD: 'YES'
CHGRP_CMD: 'chgrp rstprod'
HPSSARCH: 'YES'
HPSS_PROJECT: emc-global
LOCALARCH: 'NO'
ATARDIR: '/NCEPDEV/${HPSS_PROJECT}/1year/${USER}/${machine}/scratch/${PSLOT}'
MAKE_NSSTBUFR: 'NO'
Expand Down
1 change: 1 addition & 0 deletions workflow/hosts/jet.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ PARTITION_SERVICE: service
CHGRP_RSTPROD: 'YES'
CHGRP_CMD: 'chgrp rstprod'
HPSSARCH: 'YES'
HPSS_PROJECT: emc-global
LOCALARCH: 'NO'
ATARDIR: '/NCEPDEV/${HPSS_PROJECT}/1year/${USER}/${machine}/scratch/${PSLOT}'
MAKE_NSSTBUFR: 'NO'
Expand Down
1 change: 1 addition & 0 deletions workflow/hosts/orion.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ PARTITION_SERVICE: service
CHGRP_RSTPROD: 'YES'
CHGRP_CMD: 'chgrp rstprod'
HPSSARCH: 'NO'
HPSS_PROJECT: emc-global
LOCALARCH: 'NO'
ATARDIR: '${NOSCRUB}/archive_rotdir/${PSLOT}'
MAKE_NSSTBUFR: 'NO'
Expand Down
1 change: 1 addition & 0 deletions workflow/hosts/s4.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ PARTITION_SERVICE: serial
CHGRP_RSTPROD: 'NO'
CHGRP_CMD: 'ls'
HPSSARCH: 'NO'
HPSS_PROJECT: emc-global
LOCALARCH: 'NO'
ATARDIR: '${NOSCRUB}/archive_rotdir/${PSLOT}'
MAKE_NSSTBUFR: 'YES'
Expand Down
1 change: 1 addition & 0 deletions workflow/hosts/wcoss2.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -16,6 +16,7 @@ PARTITION_SERVICE: ''
CHGRP_RSTPROD: 'YES'
CHGRP_CMD: 'chgrp rstprod'
HPSSARCH: 'NO'
HPSS_PROJECT: emc-global
LOCALARCH: 'NO'
ATARDIR: '/NCEPDEV/${HPSS_PROJECT}/1year/${USER}/${machine}/scratch/${PSLOT}'
MAKE_NSSTBUFR: 'NO'
Expand Down
11 changes: 11 additions & 0 deletions workflow/setup_expt.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -235,6 +235,8 @@ def update_configs(host, inputs):

# loop over other configs and update them
for cfg in yaml_dict.keys():
if cfg == 'base':
continue
cfg_file = f'{inputs.expdir}/{inputs.pslot}/config.{cfg}'
cfg_dict = get_template_dict(yaml_dict[cfg])
edit_config(cfg_file, cfg_file, cfg_dict)
Expand Down Expand Up @@ -289,6 +291,15 @@ def edit_baseconfig(host, inputs):
extend_dict = {"@CCPP_SUITE@": "FV3_GFS_v17_p8", "@IMP_PHYSICS@": 8}
tmpl_dict = dict(tmpl_dict, **extend_dict)

# Read in the YAML file to fill out templates and override host defaults
yaml_path = inputs.yaml
yaml_dict = YAMLFile(path=yaml_path)
try:
cfg_dict = get_template_dict(yaml_dict['base'])
except KeyError:
cfg_dict = {}
tmpl_dict = dict(tmpl_dict, **cfg_dict)

@aerorahul aerorahul May 11, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Suggested change
# Read in the YAML file to fill out templates and override host defaults
yaml_path = inputs.yaml
yaml_dict = YAMLFile(path=yaml_path)
try:
cfg_dict = get_template_dict(yaml_dict['base'])
except KeyError:
cfg_dict = {}
tmpl_dict = dict(tmpl_dict, **cfg_dict)
try:
tmpl_dict = dict(tmpl_dict, **get_template_dict(yaml_dict['base']))
except KeyError:
pass

@aerorahul aerorahul May 11, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In line 247, can you accept yaml_dict?

def edit_baseconfig(host, inputs, yaml_dict):

@aerorahul aerorahul May 11, 2023

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, move lines 233, 234 before 231 and update 231 to pass yaml_dict?

    yaml_path = inputs.yaml
    yaml_dict = YAMLFile(path=yaml_path)

    edit_baseconfig(host, inputs, yaml_dict)

@CoryMartin-NOAA CoryMartin-NOAA May 11, 2023

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ok yeah sure

base_input = f'{inputs.configdir}/config.base.emc.dyn'
base_output = f'{inputs.expdir}/{inputs.pslot}/config.base'
edit_config(base_input, base_output, tmpl_dict)
Expand Down