Skip to content

[develop] Incorporate RRFS_dev1 workflow to SRW and import most tasks#540

Closed
danielabdi-noaa wants to merge 163 commits into
ufs-community:developfrom
danielabdi-noaa:feature/process_bufr
Closed

[develop] Incorporate RRFS_dev1 workflow to SRW and import most tasks#540
danielabdi-noaa wants to merge 163 commits into
ufs-community:developfrom
danielabdi-noaa:feature/process_bufr

Conversation

@danielabdi-noaa
Copy link
Copy Markdown
Collaborator

@danielabdi-noaa danielabdi-noaa commented Jan 16, 2023 •
edited
Loading

DESCRIPTION OF CHANGES:

I am opening up this PR for feedback per our discussion today. Will keep on working on it or a second PR but it is probably time
to get feedback on it. The tasks for both spinup (cold and warm start) and production cycles pass on CONUS domain 3km resolution as shown below. Although I am pretty sure there is a lot more work to do with other test cases I haven't considered, the new code that is constantly coming from RRFS_dev1 etc. So this is just a start to get atleast a test case running partially and try to keep up with the changes from RRFS_dev1.

It should pass fundamental tests now -- tested on Hera successfully.

Addresses the main issue of incorporating RRFS_dev1 workflow

And also adds the following tasks that have successfully passed for the specific test case I considered

The run_archive test although imported can only be tested with someone with write privilege on HPSS.

Tagging RRFS_dev1 developers for feedback: @hu5970 @EricJames-NOAA @chunhuazhou @ShunLiu-NOAA @TrevorAlcott-NOAA @guoqing-noaa @hongli-wang @terraladwig @daviddowellNOAA

Result

Result on the config.rrfs.yaml file for the CONUS domain on 3km resolution

       CYCLE                    TASK                       JOBID               STATE         EXIT STATUS     TRIES      DURATION
================================================================================================================================
202107240000          get_extrn_lbcs                    41445542           SUCCEEDED                   0         1          20.0
202107240000            make_lbcs_00                    41445663           SUCCEEDED                   0         1         154.0
202107240000            make_lbcs_01                    41445664           SUCCEEDED                   0         1         151.0
202107240000            make_lbcs_02                    41445665           SUCCEEDED                   0         1         154.0
202107240000            make_lbcs_03                    41445667           SUCCEEDED                   0         1         151.0
202107240000            make_lbcs_04                    41445668           SUCCEEDED                   0         1         152.0
202107240000            make_lbcs_05                    41445669           SUCCEEDED                   0         1         152.0
202107240000            make_lbcs_06                    41445670           SUCCEEDED                   0         1         155.0
202107240000            make_lbcs_07                    41445671           SUCCEEDED                   0         1         152.0
202107240000            make_lbcs_08                    41445673           SUCCEEDED                   0         1         151.0
202107240000            make_lbcs_09                    41445674           SUCCEEDED                   0         1          89.0
================================================================================================================================
202107240300    process_radarref_spinup                    41452897           SUCCEEDED                   0         1         185.0
202107240300     process_bufr_spinup                    41445544           SUCCEEDED                   0         1          76.0
202107240300           get_extrn_ics                    41445545           SUCCEEDED                   0         1          16.0
202107240300                make_ics                    41445676           SUCCEEDED                   0         1         135.0
202107240300         prep_cyc_spinup                    41445872           SUCCEEDED                   0         1          36.0
202107240300    anal_gsi_input_spinup                    41445873           SUCCEEDED                   0         1         347.0
202107240300    postanal_input_spinup                    41445922           SUCCEEDED                   0         1          21.0
202107240300    cldanl_nonvar_spinup                    41445926           SUCCEEDED                   0         1          93.0
202107240300         run_fcst_spinup                    41445929           SUCCEEDED                   0         1         349.0
202107240300    save_restart_spinup_f001                    41446035           SUCCEEDED                   0         1          33.0
202107240300    run_post_spinup_f000                    41471267           SUCCEEDED                   0         1         126.0
202107240300    run_prdgen_spinup_f000                    41471274           SUCCEEDED                   0         1          38.0
202107240300    run_post_spinup_f001                    41471272           SUCCEEDED                   0         1         115.0
202107240300    run_prdgen_spinup_f001                    41471275           SUCCEEDED                   0         1          38.0
================================================================================================================================
202107240400    process_radarref_spinup                    41452898           SUCCEEDED                   0         1         187.0
202107240400     process_bufr_spinup                    41445547           SUCCEEDED                   0         1          71.0
202107240400         prep_cyc_spinup                    41446051           SUCCEEDED                   0         1          66.0
202107240400    anal_gsi_input_spinup                    41446060           SUCCEEDED                   0         1         314.0
202107240400    postanal_input_spinup                    41446117           SUCCEEDED                   0         1          35.0
202107240400    cldanl_nonvar_spinup                    41446126           SUCCEEDED                   0         1          96.0
202107240400         run_fcst_spinup                    41446148           SUCCEEDED                   0         1         356.0
202107240400    save_restart_spinup_f001                    41446238           SUCCEEDED                   0         1          54.0
202107240400    run_post_spinup_f000                    41471295           SUCCEEDED                   0         1         121.0
202107240400    run_prdgen_spinup_f000                    41471338           SUCCEEDED                   0         1          40.0
202107240400    run_post_spinup_f001                    41471296           SUCCEEDED                   0         1         117.0
202107240400    run_prdgen_spinup_f001                    41471339           SUCCEEDED                   0         1          40.0
================================================================================================================================
202107240500    process_radarref_spinup                    41452899           SUCCEEDED                   0         1         186.0
202107240500     process_bufr_spinup                    41445549           SUCCEEDED                   0         1          75.0
202107240500         prep_cyc_spinup                    41446248           SUCCEEDED                   0         1          59.0
202107240500    anal_gsi_input_spinup                    41451947           SUCCEEDED                   0         1         347.0
202107240500    postanal_input_spinup                    41452170           SUCCEEDED                   0         1          41.0
202107240500    cldanl_nonvar_spinup                    41452303           SUCCEEDED                   0         1          97.0
202107240500         run_fcst_spinup                    41452373           SUCCEEDED                   0         1         362.0
202107240500    save_restart_spinup_f001                    41452611           SUCCEEDED                   0         1          57.0
202107240500    run_post_spinup_f000                    41471297           SUCCEEDED                   0         1         120.0
202107240500    run_prdgen_spinup_f000                    41471340           SUCCEEDED                   0         1          40.0
202107240500    run_post_spinup_f001                    41471298           SUCCEEDED                   0         1         117.0
202107240500    run_prdgen_spinup_f001                    41471341           SUCCEEDED                   0         1          41.0
================================================================================================================================
202107240600    process_radarref_spinup                    41452900           SUCCEEDED                   0         1         187.0
202107240600     process_bufr_spinup                    41445551           SUCCEEDED                   0         1          72.0
202107240600          get_extrn_lbcs                    41445552           SUCCEEDED                   0         1          19.0
202107240600            make_lbcs_00                    41445677           SUCCEEDED                   0         1         152.0
202107240600            make_lbcs_01                    41445678           SUCCEEDED                   0         1         149.0
202107240600            make_lbcs_02                    41445679           SUCCEEDED                   0         1         152.0
202107240600            make_lbcs_03                    41445680           SUCCEEDED                   0         1         153.0
202107240600            make_lbcs_04                    41445682           SUCCEEDED                   0         1         152.0
202107240600            make_lbcs_05                    41445683           SUCCEEDED                   0         1         156.0
202107240600            make_lbcs_06                    41445684           SUCCEEDED                   0         1         151.0
202107240600            make_lbcs_07                    41445685           SUCCEEDED                   0         1         151.0
202107240600            make_lbcs_08                    41445686           SUCCEEDED                   0         1         151.0
202107240600            make_lbcs_09                    41445690           SUCCEEDED                   0         1          88.0
202107240600         prep_cyc_spinup                    41452730           SUCCEEDED                   0         1          64.0
202107240600    anal_gsi_input_spinup                    41452770           SUCCEEDED                   0         1         339.0
202107240600    postanal_input_spinup                    41452827           SUCCEEDED                   0         1          38.0
202107240600    cldanl_nonvar_spinup                    41452865           SUCCEEDED                   0         1         103.0
202107240600         run_fcst_spinup                    41452901           SUCCEEDED                   0         1         380.0
202107240600    save_restart_spinup_f001                    41453899           SUCCEEDED                   0         1          63.0
202107240600    run_post_spinup_f000                    41471299           SUCCEEDED                   0         1         119.0
202107240600    run_prdgen_spinup_f000                    41471342           SUCCEEDED                   0         1          42.0
202107240600    run_post_spinup_f001                    41471300           SUCCEEDED                   0         1         117.0
202107240600    run_prdgen_spinup_f001                    41471343           SUCCEEDED                   0         1          42.0
================================================================================================================================
202107240700    process_radarref_spinup                    41452902           SUCCEEDED                   0         1         185.0
202107240700     process_bufr_spinup                    41445804           SUCCEEDED                   0         1          78.0
202107240700         prep_cyc_spinup                    41453977           SUCCEEDED                   0         1          63.0
202107240700    anal_gsi_input_spinup                    41454107           SUCCEEDED                   0         1         343.0
202107240700    postanal_input_spinup                    41454316           SUCCEEDED                   0         1          40.0
202107240700    cldanl_nonvar_spinup                    41454443           SUCCEEDED                   0         1         101.0
202107240700         run_fcst_spinup                    41454645           SUCCEEDED                   0         1         368.0
202107240700    save_restart_spinup_f001                    41456703           SUCCEEDED                   0         1          73.0
202107240700    run_post_spinup_f000                    41471301           SUCCEEDED                   0         1         123.0
202107240700    run_prdgen_spinup_f000                    41471345           SUCCEEDED                   0         1          43.0
202107240700    run_post_spinup_f001                    41471302           SUCCEEDED                   0         1         121.0
202107240700    run_prdgen_spinup_f001                    41471344           SUCCEEDED                   0         1          43.0
================================================================================================================================
202107240800    process_radarref_spinup                    41468567           SUCCEEDED                   0         1         187.0
202107240800     process_bufr_spinup                    41468659           SUCCEEDED                   0         1          85.0
202107240800         prep_cyc_spinup                    41468660           SUCCEEDED                   0         1          71.0
202107240800    anal_gsi_input_spinup                    41468706           SUCCEEDED                   0         1         347.0
202107240800    postanal_input_spinup                    41469056           SUCCEEDED                   0         1          41.0
202107240800    cldanl_nonvar_spinup                    41469064           SUCCEEDED                   0         1         102.0
202107240800         run_fcst_spinup                    41469105           SUCCEEDED                   0         1         394.0
202107240800    save_restart_spinup_f001                    41469343           SUCCEEDED                   0         1          65.0
202107240800    run_post_spinup_f000                    41471308           SUCCEEDED                   0         1         121.0
202107240800    run_prdgen_spinup_f000                    41471346           SUCCEEDED                   0         1          41.0
202107240800    run_post_spinup_f001                    41471309           SUCCEEDED                   0         1         120.0
202107240800    run_prdgen_spinup_f001                    41471347           SUCCEEDED                   0         1          43.0
================================================================================================================================
202107240900    process_radarref_prod                    41469408           SUCCEEDED                   0         1         192.0
202107240900       process_bufr_prod                    41469477           SUCCEEDED                   0         1          86.0
202107240900           prep_cyc_prod                    41469594           SUCCEEDED                   0         1          72.0
202107240900     anal_gsi_input_prod                    41469742           SUCCEEDED                   0         1         372.0
202107240900     postanal_input_prod                    41469892           SUCCEEDED                   0         1          45.0
202107240900      cldanl_nonvar_prod                    41469937           SUCCEEDED                   0         1         104.0
202107240900           run_fcst_prod                    41469957           SUCCEEDED                   0         1         625.0
202107240900    save_restart_prod_f1                    41470289           SUCCEEDED                   0         1          71.0
202107240900           run_post_f000                    41470837           SUCCEEDED                   0         1         120.0
202107240900           run_post_f001                    41471174           SUCCEEDED                   0         1         120.0
202107240900           run_post_f002                    41471175           SUCCEEDED                   0         1         124.0
202107240900           run_post_f003                    41471176           SUCCEEDED                   0         1         122.0
202107240900         run_prdgen_f000                    41470957           SUCCEEDED                   0         1          37.0
202107240900         run_prdgen_f001                    41471205           SUCCEEDED                   0         1          38.0
202107240900         run_prdgen_f002                    41471206           SUCCEEDED                   0         1          37.0
202107240900         run_prdgen_f003                    41471207           SUCCEEDED                   0         1          38.0
202107240900        python_maps_full                    41554794           SUCCEEDED                   0         1         273.0
202107240900           python_skewts                    41554795           SUCCEEDED                   0         1         298.0
202107240900               run_clean                    41509831           SUCCEEDED                   0         1          32.0
================================================================================================================================

Plots

Cool graphics from pygraf not the one we already have in SRW i.e.plot_allvars.
Some "colorful" ones:

wspeed_full_10m_f000
temp_full_sfc_f000
pres_full_sfc_f000
hlcy_full_sr01_f000

Type of change

  • Bug fix (non-breaking change which fixes an issue)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • This change requires a documentation update

TESTS CONDUCTED:

  • hera.intel
  • orion.intel
  • cheyenne.intel
  • cheyenne.gnu
  • gaea.intel
  • jet.intel
  • wcoss2.intel
  • NOAA Cloud (indicate which platform)
  • Jenkins
  • fundamental test suite
  • comprehensive tests (specify which if a subset was used)

DEPENDENCIES:

To be investigated

DOCUMENTATION:

Definately needs a lot of documentation once this PR is ready.

ISSUE:

CHECKLIST

  • My code follows the style guidelines in the Contributor's Guide
  • I have performed a self-review of my own code using the Code Reviewer's Guide
  • I have commented my code, particularly in hard-to-understand areas
  • My changes need updates to the documentation. I have made corresponding changes to the documentation
  • My changes do not require updates to the documentation (explain).
  • My changes generate no new warnings
  • New and existing tests pass with my changes
  • Any dependent changes have been merged and published

LABELS (optional):

A Code Manager needs to add the following labels to this PR:

  • Work In Progress
  • bug
  • enhancement
  • documentation
  • release
  • high priority
  • run_ci
  • run_we2e_fundamental_tests
  • run_we2e_comprehensive_tests
  • Needs Cheyenne test
  • Needs Jet test
  • Needs Hera test
  • Needs Orion test
  • help wanted

CONTRIBUTORS (optional):

@danielabdi-noaa danielabdi-noaa changed the title Feature/process bufr [develop] Incorporate RRFS_dev1 workflow to SRW and import few tasks Jan 16, 2023
@danielabdi-noaa danielabdi-noaa force-pushed the feature/process_bufr branch 6 times, most recently from 605ad43 to ae65ee5 Compare January 23, 2023 06:51
@danielabdi-noaa danielabdi-noaa changed the title [develop] Incorporate RRFS_dev1 workflow to SRW and import few tasks [develop] Incorporate RRFS_dev1 workflow to SRW and import most tasks Jan 24, 2023
@danielabdi-noaa danielabdi-noaa marked this pull request as ready for review January 24, 2023 04:05
This was referenced Mar 8, 2023
Closed
Closed
@danielabdi-noaa danielabdi-noaa mentioned this pull request Mar 10, 2023
Merged
37 tasks
@natalie-perlin natalie-perlin mentioned this pull request Sep 9, 2023
Merged
23 tasks
@MichaelLueken
Copy link
Copy Markdown
Collaborator

MichaelLueken commented Sep 11, 2023

With the removal of RRFS-related features (please see PR #893), it is safe to close this PR now.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mkavulich mkavulich Awaiting requested review from mkavulich

@gsketefian gsketefian Awaiting requested review from gsketefian gsketefian is a code owner

@JeffBeck-NOAA JeffBeck-NOAA Awaiting requested review from JeffBeck-NOAA JeffBeck-NOAA is a code owner

@RatkoVasic-NOAA RatkoVasic-NOAA Awaiting requested review from RatkoVasic-NOAA RatkoVasic-NOAA is a code owner

@BenjaminBlake-NOAA BenjaminBlake-NOAA Awaiting requested review from BenjaminBlake-NOAA

@ywangwof ywangwof Awaiting requested review from ywangwof ywangwof is a code owner

@chan-hoo chan-hoo Awaiting requested review from chan-hoo chan-hoo is a code owner

@panll panll Awaiting requested review from panll panll is a code owner

@christinaholtNOAA christinaholtNOAA Awaiting requested review from christinaholtNOAA christinaholtNOAA is a code owner

@christopherwharrop-noaa christopherwharrop-noaa Awaiting requested review from christopherwharrop-noaa

@mark-a-potts mark-a-potts Awaiting requested review from mark-a-potts

@jkbk2004 jkbk2004 Awaiting requested review from jkbk2004 jkbk2004 is a code owner

@willmayfield willmayfield Awaiting requested review from willmayfield

@dmwright526 dmwright526 Awaiting requested review from dmwright526

@gspetro-NOAA gspetro-NOAA Awaiting requested review from gspetro-NOAA gspetro-NOAA is a code owner

@natalie-perlin natalie-perlin Awaiting requested review from natalie-perlin natalie-perlin is a code owner

@EdwardSnyder-NOAA EdwardSnyder-NOAA Awaiting requested review from EdwardSnyder-NOAA EdwardSnyder-NOAA is a code owner

@MichaelLueken MichaelLueken Awaiting requested review from MichaelLueken MichaelLueken is a code owner

Assignees

No one assigned

Labels

Work in Progress

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Incorporate all RRFS_dev1 variables and workflow configurations

8 participants

@danielabdi-noaa @venitahagerty @hu5970 @panll @JeffBeck-NOAA @MichaelLueken @EdwardSnyder-NOAA @chan-hoo