Porting UPP for gfs v16 on wcoss2 by WenMeng-NOAA · Pull Request #420 · NOAA-EMC/UPP

WenMeng-NOAA · 2021-12-02T18:30:50Z

The changes are added for porting UPP for gfs v16 on wcoss2.

building upp lib.

post_wcoss2.lua.

aerorahul · 2021-12-02T20:54:06Z

@WenMeng-NOAA why are manage_externals changes required here?

WenMeng-NOAA · 2021-12-02T21:07:32Z

@aerorahul @KateFriedman-NOAA found a new version of manage_externals is needed for the global-workflow checking gtg code.

KateFriedman-NOAA · 2021-12-03T16:09:20Z

@aerorahul @KateFriedman-NOAA found a new version of manage_externals is needed for the global-workflow checking gtg code.

@WenMeng-NOAA Can the GTG checkout happen without manage_externals? I use whatever method you provide on the workflow side to do the checkouts. We're trying to move away from manage_externals now for a few reasons. Can we clone the GTG with submodule commands instead? Thanks!

KateFriedman-NOAA

Non-manage-externals-related changes look good and cover the changes Wen and I discussed when I initially tested the post jobs on WCOSS2. I am rebuilding my copy of the UPP with the recent LUA updates and will do the following for this PR:

retest post jobs on Dogwood with updated UPP copy and have Wen look at output
update my copy of the UPP to use submodules instead of manage_externals; will provide needed updates for Wen to incorporate

Stay tuned...

WenMeng-NOAA · 2021-12-03T16:21:47Z

@aerorahul @KateFriedman-NOAA found a new version of manage_externals is needed for the global-workflow checking gtg code.

@WenMeng-NOAA Can the GTG checkout happen without manage_externals? I use whatever method you provide on the workflow side to do the checkouts. We're trying to move away from manage_externals now for a few reasons. Can we clone the GTG with submodule commands instead? Thanks!

@KateFriedman-NOAA Yes, @aerorahul has been helping me on configurating gtg code checking out as a submodule.

KateFriedman-NOAA · 2021-12-03T16:23:18Z

@WenMeng-NOAA Please see the following log from rebuilding the UPP after recent updates, please confirm the build went as intended:

/lfs/h2/emc/global/noscrub/Kate.Friedman/git/feature-ops-wcoss2/sorc/logs/build_ncep_post_test5.log

Thanks!

Note: this is before I make any changes for the GTG checkout

WenMeng-NOAA · 2021-12-03T16:32:44Z

@WenMeng-NOAA Please see the following log from rebuilding the UPP after recent updates, please confirm the build went as intended:

/lfs/h2/emc/global/noscrub/Kate.Friedman/git/feature-ops-wcoss2/sorc/logs/build_ncep_post_test5.log

Thanks!

Note: this is before I make any changes for the GTG checkout

@KateFriedman-NOAA Your build log look good to me. Thanks!

KateFriedman-NOAA · 2021-12-03T16:34:16Z

@KateFriedman-NOAA Your build log look good to me. Thanks!

Thanks for confirming @WenMeng-NOAA ! I'm working on the patch to retire manage_externals and use submodules instead for the GTG checkout. Will provide the patch to include in this PR when ready.

KateFriedman-NOAA · 2021-12-03T17:50:09Z

@WenMeng-NOAA A quick update... @aerorahul and I are still working on the patch (ironing out a detail) but I have removed manage_externals in my fork copy of the post_gfsv16_wcoss2 branch and added the initial submodule setup. See my changes thus far at this link:

WenMeng-NOAA/UPP@post_gfsv16_wcoss2...KateFriedman-NOAA:post_gfsv16_wcoss2

I hacked things on the workflow side a tad to get the right tag version of the GTG checked out (the detail to iron out) and tested the build with GTG changes in place. Please see the following log and confirm that my build with GTG worked:

/lfs/h2/emc/global/noscrub/Kate.Friedman/git/feature-ops-wcoss2/sorc/logs/build_ncep_post_test6_gtg.log

I will submit a PR into your fork post_gfsv16_wcoss2 branch with the patch when ready. Thanks!

WenMeng-NOAA · 2021-12-03T18:04:01Z

/lfs/h2/emc/global/noscrub/Kate.Friedman/git/feature-ops-wcoss2/sorc/logs/build_ncep_post_test6_gtg.log

Your build log looks good to me. Thanks!

KateFriedman-NOAA · 2021-12-03T18:13:06Z

Your build log looks good to me. Thanks!

Thanks @WenMeng-NOAA ! I just reran all of my gdaspost* jobs and the gfspost023 job in my Dogwood test. Please see the following logs and output and let me know if it looks ok:

gdaspost*:
/lfs/h2/emc/ptmp/Kate.Friedman/comrot/testcyc768b/logs/2021082400/gdaspost0*log
/lfs/h2/emc/ptmp/Kate.Friedman/comrot/testcyc768b/gdas.20210824/00/atmos

gfspost023 (just redid the f213 files):
/lfs/h2/emc/ptmp/Kate.Friedman/comrot/testcyc768b/logs/2021082400/gfspost023.log
/lfs/h2/emc/ptmp/Kate.Friedman/comrot/testcyc768b/gfs.20210824/00/atmos/f213

FYI, I will be running a cleaner fresh test next week for everyone to examine, so I'll send logs/output to look at again when I do that test. Thanks!

WenMeng-NOAA · 2021-12-06T15:24:09Z

Your build log looks good to me. Thanks!

Thanks @WenMeng-NOAA ! I just reran all of my gdaspost* jobs and the gfspost023 job in my Dogwood test. Please see the following logs and output and let me know if it looks ok:

gdaspost*: /lfs/h2/emc/ptmp/Kate.Friedman/comrot/testcyc768b/logs/2021082400/gdaspost0*log /lfs/h2/emc/ptmp/Kate.Friedman/comrot/testcyc768b/gdas.20210824/00/atmos

gfspost023 (just redid the f213 files): /lfs/h2/emc/ptmp/Kate.Friedman/comrot/testcyc768b/logs/2021082400/gfspost023.log /lfs/h2/emc/ptmp/Kate.Friedman/comrot/testcyc768b/gfs.20210824/00/atmos/f213

FYI, I will be running a cleaner fresh test next week for everyone to examine, so I'll send logs/output to look at again when I do that test. Thanks!

@KateFriedman-NOAA The runtime logs and UPP datasets look good to me. Thanks!

KateFriedman-NOAA · 2021-12-06T19:08:26Z

@WenMeng-NOAA I submitted PR #5 from my fork to your fork with updates for converting the GTG checkout from manage_externals to git submodules. I wasn't able to add reviewers or assignees or labels.

WenMeng-NOAA · 2021-12-07T14:21:44Z

@KateFriedman-NOAA Your PR looks good to me. Would you like to have someone else review it?

KateFriedman-NOAA · 2021-12-07T14:29:54Z

@KateFriedman-NOAA Your PR looks good to me. Would you like to have someone else review it?

Nope, since it works for you and both @aerorahul and myself tested it, it should be good to submit my PR into your branch. Thanks!

* Retire manage_externals for GTG checkout - remove manage_externals copy from repo - remove associated Externals.cfg file - replacing GTG checkout with submodules Refs: NOAA-EMC#363 * Add UPP_GTG submodule - created .gitmodules file via git submodule add command: git submodule add https://github.com/NCAR/UPP_GTG sorc/post_gtg.fd - new post_gtg.fd subfolder added (empty before submodule checkout) Refs: NOAA-EMC#363 * Updating submodule pointer of GTG to ncep_post_gtg.v2.0.4

WenMeng-NOAA · 2021-12-09T15:34:13Z

@KateFriedman-NOAA and @aerorahul It looks the branch is working for inline and offline post in Kate's GFS V16 end-to-end testing on WCOSS2. Do you have more comments on it? Thanks!

KateFriedman-NOAA

Looks good and post jobs in GFS tests are working as intended. Thanks!

aerorahul

Defer to @KateFriedman-NOAA. No further comments from me.

KateFriedman-NOAA · 2021-12-09T16:14:28Z

@WenMeng-NOAA If you could look at the post output in my free-forecast test that ran overnight and then, assuming it looks good, I'm good for this to be submitted. See this g-w issue comment with output details: NOAA-EMC/global-workflow#399 (comment) . Thanks!

Wen Meng and others added 11 commits September 7, 2021 13:26

Add changes for building executable on wcoss2.

59b755b

Add changes for buding upp lib on wcoss2.

5c1ea4f

Add changes for building upp lib on wcoss2.

0059e1d

Update hdf5 and netcdf modules.

0b1cb80

Upgrade manage_externals to new version which works for python 3.8.5.

bcf004f

Remove jlogfile in workflow scripts.

d030a7c

Changes for overriding module versions from upstream setting.

d01f8b7

Changes for overriding module versions from upstream setting for

3e1e8fb

building upp lib.

Update for POSTGRB2TBL default setting.

47eecdb

Update for LMod compliance.

40e2a9f

Remove v8.0.0-wcoss2 and lib-wcoss2; correct typo in upp_wcoss2.lua and

9846295

post_wcoss2.lua.

WenMeng-NOAA added the GFSV16 label Dec 2, 2021

WenMeng-NOAA requested a review from KateFriedman-NOAA December 2, 2021 18:31

WenMeng-NOAA linked an issue Dec 2, 2021 that may be closed by this pull request

Port UPP for GFS V16 to WCOSS2 #363

Closed

WenMeng-NOAA requested a review from aerorahul December 3, 2021 13:47

KateFriedman-NOAA reviewed Dec 3, 2021

View reviewed changes

Comment thread jobs/J_NCEPPOST Outdated

KateFriedman-NOAA suggested changes Dec 3, 2021

View reviewed changes

Remove jflofile per Kate's comments

afb4d9c

KateFriedman-NOAA mentioned this pull request Dec 6, 2021

Change GTG checkout to submodule and remove manage_externals WenMeng-NOAA/UPP#5

Merged

WenMeng-NOAA added the Ready for Review This PR is ready for code review. label Dec 7, 2021

Teak makefile_lib.

7df76e6

KateFriedman-NOAA approved these changes Dec 9, 2021

View reviewed changes

aerorahul approved these changes Dec 9, 2021

View reviewed changes

WenMeng-NOAA merged commit 52af7f9 into NOAA-EMC:release/gfs_v16 Dec 9, 2021

WenMeng-NOAA mentioned this pull request Dec 9, 2021

Updates in branch release/gfs_v16 #165

Closed

WenMeng-NOAA deleted the post_gfsv16_wcoss2 branch January 10, 2022 17:22

EricJames-NOAA pushed a commit to EricJames-NOAA/UPP that referenced this pull request Dec 14, 2022

update the fix file: RRFS_NA_3km/regional_rrfsfv3_profdat (NOAA-EMC#420)

4c19262

Conversation

WenMeng-NOAA commented Dec 2, 2021

Uh oh!

aerorahul commented Dec 2, 2021

Uh oh!

WenMeng-NOAA commented Dec 2, 2021

Uh oh!

KateFriedman-NOAA commented Dec 3, 2021

Uh oh!

Uh oh!

KateFriedman-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

WenMeng-NOAA commented Dec 3, 2021

Uh oh!

KateFriedman-NOAA commented Dec 3, 2021

Uh oh!

WenMeng-NOAA commented Dec 3, 2021

Uh oh!

KateFriedman-NOAA commented Dec 3, 2021

Uh oh!

KateFriedman-NOAA commented Dec 3, 2021

Uh oh!

WenMeng-NOAA commented Dec 3, 2021

Uh oh!

KateFriedman-NOAA commented Dec 3, 2021

Uh oh!

WenMeng-NOAA commented Dec 6, 2021

Uh oh!

KateFriedman-NOAA commented Dec 6, 2021

Uh oh!

WenMeng-NOAA commented Dec 7, 2021

Uh oh!

KateFriedman-NOAA commented Dec 7, 2021

Uh oh!

WenMeng-NOAA commented Dec 9, 2021

Uh oh!

KateFriedman-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

KateFriedman-NOAA commented Dec 9, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants