Add the CMEPS based HAFS-HYCOM coupling into the HAFS workflow#38
Conversation
Sync feature/hafs_couplehycom with develop
…s_couplehycom branch.
…20' into feature/hafs_couplehycom
feature/hafs_couplehycom branch, which has the CMEPS based HYCOM coupling capabilities. B. Update the workflow to support the CMEPS-based HYCOM coupling
forecast_resource entity on Orion and Hera. B. Update the indentation in nems.configure*.tmp files
…ready to be merged back into the develop branch.
BinLiu-NOAA
requested review from
BijuThomas-NOAA,
ZhanZhang-NOAA,
danrosen25,
evankalina,
hyunsookkim-NOAA and
uturuncoglu
|
Hi @BinLiu-NOAA, |
Hi @danrosen25, Sound great. Please let us know once the CMEPS pull request is done. We will update this pull request to use the latest CMEPS's support/HAFS branch as well. Thanks! Bin |
…port/HAFS branch of CMEPS. And a new ccpp SDF suite_HAFS_v0_gfdlmp_nonsst.xml is added.
|
Hi @danrosen25, I saw the CMEPS Pull Request has been merged. And I have update this feature/hafs_coupledhycom branch to use the latest support/HAFS branch accordingly. In the meantime, a new CCPP suite definition file: suite_HAFS_v0_gfdlmp_nonsst.xml is added, which will be used in hafsv0.2 baseline experiment. So @uturuncoglu, @danrosen25, @BijuThomas-NOAA, @ZhanZhang-NOAA, @hyunsookkim-NOAA, @evankalina, please help to review this PR request when you get a chance. Thanks! Bin P.S, @BijuThomas-NOAA could you please post the wall clock time difference here between the two experiments you conducted with the direct coupling and CMEPS based coupling? Thanks! |
|
Here is the comparison of Coupled HAFS run with Direct Coupling versus
CMEPS Mediator
Total Tasks: 1340 (Atmos 1200, IO 80, Ocean 60)
Wall time(126h forecast):
- Direct coupling *3h 25m*
- CMEPS mediator *3h 32m (~7 min overhead)*
- Results are not bitwise identical
Detailed ESMF_Profile summary
- Orion: /work/noaa/hwrf/noscrub/bthomas/ESMF_Profile.summary
-
Hera: /scratch1/NCEPDEV/hwrf/noscrub/Biju.Thomas/ESMF_Profile.summary
…On Thu, Jan 28, 2021 at 2:53 PM Bin Liu ***@***.***> wrote:
Hi @danrosen25 <https://github.com/danrosen25>,
I saw the CMEPS Pull Request <ESCOMP/CMEPS#154>
has been merged. And I have update this feature/hafs_coupledhycom branch to
use the latest support/HAFS branch accordingly. In the meantime, a new CCPP
suite definition file: suite_HAFS_v0_gfdlmp_nonsst.xml is added, which will
be used in hafsv0.2 baseline experiment.
So @uturuncoglu <https://github.com/uturuncoglu>, @danrosen25
<https://github.com/danrosen25>, @BijuThomas-NOAA
<https://github.com/BijuThomas-NOAA>, @ZhanZhang-NOAA
<https://github.com/ZhanZhang-NOAA>, @hyunsookkim-NOAA
<https://github.com/hyunsookkim-NOAA>, @evankalina
<https://github.com/evankalina>, please help to review this PR request
when you get a chance.
Thanks!
Bin
P.S, @BijuThomas-NOAA <https://github.com/BijuThomas-NOAA> could you
please post the wall clock time difference here between the two experiments
you conducted with the direct coupling and CMEPS based coupling? Thanks!
—
You are receiving this because you were mentioned.
Reply to this email directly, view it on GitHub
<#38 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AE74C2GEDEXFFMN4EYWVQK3S4G6CVANCNFSM4WVWRDFA>
.
--
Biju Thomas
IMSG at NOAA/NWS/NCEP/EMC
Tel: (401)-284-0165
Email: Biju.Thomas@noaa.gov
|
|
I ran 5 debug queue tests on Hera. They each completed 12-15 hours of the simulation. Build / Run Configurations
Component Revisions
|
Hi @danrosen25, Thanks for conducting these tests! BTW, could you please point me to one of your coupled FV3ATM+HYCOM test run directory? I would like to check if the variables exchange between the two components successfully. Thanks! -Bin |
|
@BinLiu-NOAA @uturuncoglu Coupling with CMEPS is working because the fields are explicitly configured in nems.configure. |
Thanks, @danrosen25! Could you point me to your test dir for your experiment of "Build FV3ATM+HYCOM+CMEPS / Run FV3ATM+HYCOM"? My test with this configuration did not work properly (the SST field in FV3ATM did not come from HYCOM as expected). My test of "Build FV3ATM+HYCOM+CMEPS / Run FV3ATM+HYCOM+CMEPS", however, worked properly for the ocean coupling. I also looked at @BijuThomas-NOAA 's two tests with cpl_ocean=3 (CMEPS based coupling) and cpl_ocean=2 (direct coupling) under these two dirs: Bin |
|
@BinLiu-NOAA |
…e direct coupling when the forecast model is built with the CMEPS=Y option.
feature/hafs_couplehycom branch, which the HYCOM set_impexp_fields fix.
|
@BinLiu-NOAA And I updated the ufs-weather-model revision for this PR |
The changes to this file will work without CMEPS in the build but I fixed the issue anyway. |
|
@BinLiu-NOAA, then do you get HYCOM imports forcing variables from FV3 successfully? |
* Update nems.configure.atm_ocn.tmp for the direct coupling configuration
@hyunsookkim-NOAA, yes, with @danrosen25 's fix, we now have the HAFS FV3ATM-HYCOM coupling working properly for different coupling configurations, including for both direct coupling and CMEPS based coupling. |
@danrosen25 This feature/hafs_couplehycom branch has already been synced with the latest HAFS develop. And once this PR goes in, it will become the develop. So, no need to locally merge develop backward. |
uturuncoglu
left a comment
uturuncoglu
left a comment
There was a problem hiding this comment.
I can see that sorc/hafs_forecast.fd also bring some CDEPS related additions such as model level RT. Please note that RT test does not work with the updated version of the model yet. I'll look at it and fix it under feature/hafs_couplehycom_cdeps branch for the model.
@uturuncoglu Thanks for the note! And yes, we so far only test the standard regression tests in the ufs-weather-model level. We understand that the CDEPS related regression tests are not working. But, as you mentioned, you are working on fixing them in the feature/hafs_couplehycom_cdeps branch. |
Hi @BinLiu-NOAA, |
|
I have just synced the hafs_forecast.fd with its latest develop branch as of 02/04/2021 (no HAFS result changes). Also, this branch has synced with the latest HAFS develop branch. Appreciate it if someone can review and approve this pull request, so that we can move on to our next pull request. Thanks! |
|
All, the code changes look reasonable, but I would like to run the regression tests on Jet before approving. Is that a reasonable way to test these changes? Basically I'm looking for input on how to be an effective reviewer on these PRs ;-) |
Hi @evankalina, for this one you probably do not need to run the regression test. I was thinking of running all the regression tests on all supported platforms for our upcoming pull request to merge the feature/hafsv0.2_baseline branch (which has already synced up with this feautre/hafs_couplehycom branch) into the develop. I will initiate that pull request right after this one goes into develop. |
|
@BinLiu-NOAA OK, sounds good. I am OK with this going to develop. |
|
Thanks, @evankalina and @ChunxiZhang-NOAA! |
|
Thanks @BinLiu-NOAA @evankalina |
Add the CMEPS based HAFS-HYCOM coupling into the HAFS workflow Bring in the CMEPS based HAFS-HYCOM coupling by merging the feature/hafs_couplehycom branch into the develop branch Update hafs_forecast.fd to point to the latest support/HAFS branch, which now has the CMEPS based HYCOM coupling capability developed by @danrosen25, et al. Update the HAFS workflow to support the CMEPS-based HYCOM coupling. The cpl_ocean configuration option is to control the different coupling methods in the workflow. - 0, run side-by-side forecast without coupling - 1, direct NUOPC/ESMF coupling through nearest point regridding - 2, direct NUOPC/ESMF coupling through bilinear regridding - 3, CMEPS based coupling through bilinear regridding Add a new CCPP suite definition file: suite_HAFS_v0_gfdlmp_nonsst.xml Note: The cpl_ocean option 3 currently gets slightly different results from the cpl_ocean option 2, which is caused by the term ordering differences in CMEPS resulting in small roundoff differences that build each coupling time step. But, @danrosen25 was able to reproduce bit-for-bit results if using a different branch of ESMF and turning on some regridding options that affect term ordering. Overall, it's just the butterfly effect, with about a 10^-14 difference during each regridding call.
7 participants
Bring in the CMEPS based HAFS-HYCOM coupling by merging the feature/hafs_couplehycom branch into the develop branch
- 0, run side-by-side forecast without coupling
- 1, direct NUOPC/ESMF coupling through nearest point regridding
- 2, direct NUOPC/ESMF coupling through bilinear regridding
- 3, CMEPS based coupling through bilinear regridding
Note: The cpl_ocean option 3 currently gets slightly different results from the cpl_ocean option 2, which is caused by the term ordering differences in CMEPS resulting in small roundoff differences that build each coupling time step. But, @danrosen25 was able to reproduce bit-for-bit results if using a different branch of ESMF and turning on some regridding options that affect term ordering. Overall, it's just the butterfly effect, with about a 10^-14 difference during each regridding call.