Convert machine scripts to yaml format. by danielabdi-noaa · Pull Request #388 · ufs-community/ufs-srweather-app

danielabdi-noaa · 2022-09-30T21:42:00Z

DESCRIPTION OF CHANGES:

This PR converts machine files into yaml format, and then use them to update config_defaults.yaml once for reasons explained in issue #386. The final result is written to var_defns.sh so there won't be a need to source machine file in many ex- and other scripts.

Type of change

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

TESTS CONDUCTED:

DEPENDENCIES:

None

ISSUE:

#386

CHECKLIST

My code follows the style guidelines in the Contributor's Guide
I have performed a self-review of my own code using the Code Reviewer's Guide
I have commented my code, particularly in hard-to-understand areas
My changes need updates to the documentation. I have made corresponding changes to the documentation
My changes do not require updates to the documentation (explain).
My changes generate no new warnings
New and existing tests pass with my changes
Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

CONTRIBUTORS (optional):

@christinaholtNOAA

venitahagerty · 2022-09-30T23:01:52Z

Machine: jet
Compiler: intel
Job: WE
Repo location: /lfs1/BMC/nrtrr/rrfs_ci/autoci/pr/1073258097/20220930222016/ufs-srweather-app
Build was Successful
Rocoto jobs started
Long term tracking will be done on 9 experiments
If test failed, please make changes and add the following label back:
ci-jet-intel-WE
Experiment Succeeded on jet: grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_RRFS_v1beta
Experiment Succeeded on jet: nco_grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR
Experiment Succeeded on jet: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v15p2
Experiment Succeeded on jet: grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_RAP_suite_HRRR
Experiment Succeeded on jet: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_RAP_suite_HRRR
Experiment Succeeded on jet: grid_RRFS_CONUS_25km_ics_GSMGFS_lbcs_GSMGFS_suite_GFS_v15p2
Experiment Succeeded on jet: grid_RRFS_CONUS_25km_ics_FV3GFS_lbcs_FV3GFS_suite_GFS_v16
Experiment Succeeded on jet: grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_RRFS_v1beta
Experiment Succeeded on jet: grid_RRFS_CONUScompact_25km_ics_HRRR_lbcs_HRRR_suite_HRRR

MichaelLueken · 2022-10-03T18:33:41Z

@danielabdi-noaa While attempting to run the manual tests on Hera, I'm encountering the following failure:

Traceback (most recent call last):
  File "/scratch1/NCEPDEV/nems/Michael.Lueken/ufs-srweather-app/ush/calculate_cost.py", line 91, in <module>
    params = calculate_cost(args.cfg)
  File "/scratch1/NCEPDEV/nems/Michael.Lueken/ufs-srweather-app/ush/calculate_cost.py", line 38, in calculate_cost
    grid_params = set_gridparams_GFDLgrid(
  File "/scratch1/NCEPDEV/nems/Michael.Lueken/ufs-srweather-app/ush/set_gridparams_GFDLgrid.py", line 255, in set_gridparams_GFDLgrid
    halo_width_on_t7g = NH4 + 1
TypeError: unsupported operand type(s) for +: 'NoneType' and 'int'

Have you seen similar issues while running the WE2E tests on this machine?

dshawul · 2022-10-03T18:46:48Z

@MichaelLueken Sorry I introduced a bug in my last commit in the GFDL grid subroutine which I have fixed now.

MichaelLueken

@danielabdi-noaa Thank you very much for addressing my concern in running the fundamental WE2E tests manually. Another test was run on Hera which successfully completed. I approve of these changes.

panll · 2022-10-03T21:09:19Z

Testing error on Hera with default config.yaml (config.community.yaml). The code does not recognize "hera". Here is the error message when run ./generate_FV3LAM_wflow.py:
Generating the global experiment variable definitions file specified by
GLOBAL_VAR_DEFNS_FN:
GLOBAL_VAR_DEFNS_FN = "var_defns.sh"
Full path to this file is:
GLOBAL_VAR_DEFNS_FP = "/scratch2/BMC/fv3lam/lpan/test/review/10032022/test/expt_dirs/test_community/var_defns.sh"
For more detailed information, set DEBUG to "TRUE" in the experiment
configuration file ("config.yaml").
File "./generate_FV3LAM_wflow.py", line 1118, in
p.start()
File "/contrib/miniconda3/4.5.12/envs/regional_workflow/lib/python3.8/multiprocessing/process.py", line 121, in start
self._popen = self._Popen(self)
File "/contrib/miniconda3/4.5.12/envs/regional_workflow/lib/python3.8/multiprocessing/context.py", line 224, in _Popen
return _default_context.get_context().Process._Popen(process_obj)
File "/contrib/miniconda3/4.5.12/envs/regional_workflow/lib/python3.8/multiprocessing/context.py", line 276, in _Popen
return Popen(process_obj)
File "/contrib/miniconda3/4.5.12/envs/regional_workflow/lib/python3.8/multiprocessing/popen_fork.py", line 19, in init
self._launch(process_obj)
File "/contrib/miniconda3/4.5.12/envs/regional_workflow/lib/python3.8/multiprocessing/popen_fork.py", line 75, in _launch
code = process_obj._bootstrap(parent_sentinel=child_r)
File "/contrib/miniconda3/4.5.12/envs/regional_workflow/lib/python3.8/multiprocessing/process.py", line 313, in _bootstrap
self.run()
File "/contrib/miniconda3/4.5.12/envs/regional_workflow/lib/python3.8/multiprocessing/process.py", line 108, in run
self._target(*self._args, **self._kwargs)
File "./generate_FV3LAM_wflow.py", line 1106, in workflow_func
generate_FV3LAM_wflow()
File "./generate_FV3LAM_wflow.py", line 114, in generate_FV3LAM_wflow
setup()
File "/scratch2/BMC/fv3lam/lpan/test/review/10032022/test/ufs-srweather-app/ush/setup.py", line 2031, in setup
print_err_msg_exit(
File "/scratch2/BMC/fv3lam/lpan/test/review/10032022/test/ufs-srweather-app/ush/python_utils/print_msg.py", line 19, in print_err_msg_exit
traceback.print_stack(file=sys.stderr)
FATAL ERROR:
The variable MACHINE=hera in config_defaults.yaml or config.yaml does not have
a valid value. Possible values are:
MACHINE = ['HERA', 'ORION', 'JET', 'ODIN', 'CHEYENNE', 'STAMPEDE', 'LINUX', 'MACOS', 'NOAACLOUD', 'SINGULARITY', 'GAEA']
Exiting with nonzero status.
rm: cannot remove ‘/scratch2/BMC/fv3lam/lpan/test/review/10032022/test/ufs-srweather-app/ush/tmp’: No such file or directory
File "./generate_FV3LAM_wflow.py", line 1135, in
rm_vrfy(tmp_fp)
File "/scratch2/BMC/fv3lam/lpan/test/review/10032022/test/ufs-srweather-app/ush/python_utils/filesys_cmds_vrfy.py", line 33, in rm_vrfy
return cmd_vrfy("rm", *args)
File "/scratch2/BMC/fv3lam/lpan/test/review/10032022/test/ufs-srweather-app/ush/python_utils/filesys_cmds_vrfy.py", line 20, in cmd_vrfy
print_err_msg_exit(f'System call "{cmd}" failed.')
File "/scratch2/BMC/fv3lam/lpan/test/review/10032022/test/ufs-srweather-app/ush/python_utils/print_msg.py", line 19, in print_err_msg_exit
traceback.print_stack(file=sys.stderr)
FATAL ERROR: System call "rm /scratch2/BMC/fv3lam/lpan/test/review/10032022/test/ufs-srweather-app/ush/tmp" failed.
Exiting with nonzero status.
@danielabdi-noaa

danielabdi-noaa · 2022-10-03T21:26:35Z

@panll There was a bug with upper/lower case machine name that doesn't happen in WE2E tests. I fixed it now, but note that if you re-built hera binaries in last couple of days, it may fail during forecast.

panll · 2022-10-03T21:57:54Z

@danielabdi-noaa it works now, thanks!

danielabdi-noaa · 2022-10-03T22:47:09Z

@panll Thanks for testing. Unittest should have captured this bug but it was not returning exit code so it wrongly thought the test passed. I would like to test this PR on all machines using Jenkins, which is now close to being ready after Cheyenne issues are fixed.

willmayfield · 2022-10-05T17:50:34Z

  DOMAIN_PREGEN_BASEDIR: ""
+  #
+  #-----------------------------------------------------------------------
+  # Scritps and commands needed by workflow and tasks


typo "Scripts"

Thanks i will fix it in the other PR.

danielabdi-noaa changed the title ~~Make machine files yaml.~~ Convert machine scripts to yaml format. Sep 30, 2022

danielabdi-noaa force-pushed the feature/machine_cfg branch from 4289a26 to c09758a Compare September 30, 2022 22:06

danielabdi-noaa added ci-hera-intel-WE Kicks off automated workflow test on hera with intel ci-jet-intel-WE Kicks off automated workflow test on jet with intel labels Sep 30, 2022

venitahagerty removed ci-hera-intel-WE Kicks off automated workflow test on hera with intel ci-jet-intel-WE Kicks off automated workflow test on jet with intel labels Sep 30, 2022

ufs-community deleted a comment from venitahagerty Sep 30, 2022

danielabdi-noaa added the ci-hera-intel-WE Kicks off automated workflow test on hera with intel label Sep 30, 2022

venitahagerty removed the ci-hera-intel-WE Kicks off automated workflow test on hera with intel label Oct 1, 2022

danielabdi-noaa force-pushed the feature/machine_cfg branch from c09758a to 1db0fe0 Compare October 1, 2022 00:59

ufs-community deleted a comment from venitahagerty Oct 1, 2022

danielabdi-noaa added the ci-hera-intel-WE Kicks off automated workflow test on hera with intel label Oct 1, 2022

venitahagerty removed the ci-hera-intel-WE Kicks off automated workflow test on hera with intel label Oct 1, 2022

danielabdi-noaa force-pushed the feature/machine_cfg branch from 1db0fe0 to 6b266e6 Compare October 1, 2022 12:14

ufs-community deleted a comment from venitahagerty Oct 1, 2022

danielabdi-noaa added the ci-hera-intel-WE Kicks off automated workflow test on hera with intel label Oct 1, 2022

venitahagerty removed the ci-hera-intel-WE Kicks off automated workflow test on hera with intel label Oct 1, 2022

ufs-community deleted a comment from venitahagerty Oct 1, 2022

Make machine files yaml.

f3a8c3a

danielabdi-noaa force-pushed the feature/machine_cfg branch from 6b266e6 to f3a8c3a Compare October 1, 2022 15:37

danielabdi-noaa marked this pull request as ready for review October 1, 2022 15:42

danielabdi-noaa requested review from BenjaminBlake-NOAA, JeffBeck-NOAA, RatkoVasic-NOAA, chan-hoo, gsketefian, mkavulich, panll and ywangwof as code owners October 1, 2022 15:42

danielabdi-noaa requested review from EdwardSnyder-NOAA, MichaelLueken, dmwright526, gspetro-NOAA, natalie-perlin and willmayfield as code owners October 1, 2022 15:42

danielabdi-noaa added 3 commits October 2, 2022 17:49

Remove redundant SR_WX dir

f151165

Remove some duplicate derived types.

9cbeb9f

Convert constants to yaml.

1e7a2c3

danielabdi-noaa force-pushed the feature/machine_cfg branch from 20f2a96 to 1e7a2c3 Compare October 2, 2022 17:53

Bug fix GFDL grid.

003241f

MichaelLueken approved these changes Oct 3, 2022

View reviewed changes

Bug fix machine lower/upper case.

de724cd

panll approved these changes Oct 3, 2022

View reviewed changes

Fix unittest to capture exit code.

a39a7c7

danielabdi-noaa force-pushed the feature/machine_cfg branch from 57c25f4 to a39a7c7 Compare October 4, 2022 01:28

danielabdi-noaa and others added 4 commits October 3, 2022 22:15

Gaea lmod setup fix with tcsh.

6f6a193

Add missing gaea commands.

01dc89f

Remove obsolete module-setup scripts.

1f22776

Fix linux modulefile name.

db574d8

danielabdi-noaa force-pushed the feature/machine_cfg branch from c51a833 to db574d8 Compare October 4, 2022 04:38

danielabdi-noaa merged commit cc96304 into ufs-community:develop Oct 4, 2022

willmayfield reviewed Oct 5, 2022

View reviewed changes

EdwardSnyder-NOAA mentioned this pull request Nov 2, 2022

Feature/add metplus capabilities noaacloud #375

Closed

37 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Convert machine scripts to yaml format.#388

Convert machine scripts to yaml format.#388
danielabdi-noaa merged 11 commits into
ufs-community:developfrom
danielabdi-noaa:feature/machine_cfg

danielabdi-noaa commented Sep 30, 2022 •

edited

Loading

Uh oh!

venitahagerty commented Sep 30, 2022 •

edited

Loading

Uh oh!

MichaelLueken commented Oct 3, 2022

Uh oh!

dshawul commented Oct 3, 2022

Uh oh!

MichaelLueken left a comment

Uh oh!

panll commented Oct 3, 2022

Uh oh!

danielabdi-noaa commented Oct 3, 2022

Uh oh!

panll commented Oct 3, 2022

Uh oh!

danielabdi-noaa commented Oct 3, 2022

Uh oh!

willmayfield Oct 5, 2022

Uh oh!

danielabdi-noaa Oct 5, 2022

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

danielabdi-noaa commented Sep 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

DESCRIPTION OF CHANGES:

Type of change

TESTS CONDUCTED:

DEPENDENCIES:

ISSUE:

CHECKLIST

LABELS (optional):

CONTRIBUTORS (optional):

Uh oh!

venitahagerty commented Sep 30, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

MichaelLueken commented Oct 3, 2022

Uh oh!

dshawul commented Oct 3, 2022

Uh oh!

MichaelLueken left a comment

Choose a reason for hiding this comment

Uh oh!

panll commented Oct 3, 2022

Uh oh!

danielabdi-noaa commented Oct 3, 2022

Uh oh!

panll commented Oct 3, 2022

Uh oh!

danielabdi-noaa commented Oct 3, 2022

Uh oh!

willmayfield Oct 5, 2022

Choose a reason for hiding this comment

Uh oh!

danielabdi-noaa Oct 5, 2022

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

danielabdi-noaa commented Sep 30, 2022 •

edited

Loading

venitahagerty commented Sep 30, 2022 •

edited

Loading