changes to fix budget issues for aoflux on xgrid by mvertens · Pull Request #262 · ESCOMP/CMEPS

mvertens · 2021-12-25T00:44:40Z

Description of changes

Fixes to fix budget differences found between the aoflux calculation on the xgrid and aoflux calculatino on the ogrid.

Specific notes

The key difference is that mapping from the xgrid to the agrid is now done so that fields on the xgrid are normalized by the current open ocean fraction on the xgrid and then mapped and subsequently divided by the ocean fraction on the atm grid.

Contributors other than yourself, if any: None

CMEPS Issues Fixed: None

Are changes expected to change answers?

bit for bit
different at roundoff level
more substantial

Any User Interface Changes (namelist or namelist defaults changes)?

Yes
No

Testing performed if application target is CESM:(either UFS-S2S or CESM testing is required):

(recommended) CIME_DRIVER=nuopc scripts_regression_tests.py
- machines:
- details (e.g. failed tests):
(recommended) CESM testlist_drv.xml
- machines and compilers: cheyenne_intel
- details (e.g. failed tests): generated baselines for cesm2_3_alpha07c and compared to cesm2_3_alpha07c+cmeps changes in this PR.
(optional) CESM prealpha test
- machines and compilers
- details (e.g. failed tests):
(other) please described in detail
- machines and compilers
- details (e.g. failed tests):

Testing performed if application target is UFS-coupled:

(recommended) UFS-coupled testing
- description:
- details (e.g. failed tests):

Testing performed if application target is UFS-HAFS:

(recommended) UFS-HAFS testing
- description:
- details (e.g. failed tests):

Hashes used for testing:

CESM:
- repository to check out: https://github.com/ESCOMP/CESM.git
- branch:
- hash:
UFS-coupled, then umbrella repostiory to check out and associated hash:
- repository to check out: https://github.com/ufs-community/ufs-weather-model
- branch: point CMEPS to this PR (b2979b5)
- hash: ufs-weather-model hash 803e03ad
UFS-HAFS, then umbrella repostiory to check out and associated hash:
- repository to check out:
- branch:
- hash:

uturuncoglu · 2021-12-27T16:32:43Z

@mvertens is this PR ready for testing? I am plaining to test with UFS XGrid implementation but I just want to be sure.

mvertens · 2021-12-27T17:05:04Z

@ufuk @denise - this PR is ready for testing. Thanks!

DeniseWorthen

All UWM cpld/hafs/cdeps tests using CMEPS pass; hashes used for testing have been noted in the PR

uturuncoglu · 2021-12-28T16:36:59Z

@mvertens I could not test this yet since Cheyenne is not available. I could port xgrid work to Orion but it could take time. Sorry for delay.

mvertens · 2021-12-28T16:50:21Z

@uturuncoglu - no problem. Please don't take time to port this to Orion. When cheyenne comes back up you can test this.

uturuncoglu · 2021-12-30T18:38:55Z

@mvertens I tested this under UFS xgrid implementation but there is no any difference in my results when I compare it with the older version of CMEPS? Is it expected behavior? All the fluxes are same.

mvertens · 2021-12-30T18:52:00Z

@uturuncoglu - There should be diffs for Xgrid. However, It will not effect any answers that do not use Xgrid.

uturuncoglu · 2021-12-30T20:33:14Z

@mvertens it is strange. I am using xgrid but no difference. Let me check it again. Maybe there is something wrong when I merge this with my branch.

uturuncoglu · 2021-12-30T20:49:18Z

@mvertens I think I solved issue. Now merge builds without any issue but I am getting following error when I run it, i'll check it what is going on

20211230 134551.548 ERROR            PET001 ESMF_FieldRegrid.F90:784 ESMF_FieldRegridStoreNX Invalid argument  - - RegridStore on XGrid is not supported in this overloaded method
20211230 134551.548 ERROR            PET001 med_phases_aofluxes_mod.F90:845 Invalid argument  - Passing error in return code
20211230 134551.548 ERROR            PET001 med_phases_aofluxes_mod.F90:423 Invalid argument  - Passing error in return code
20211230 134551.548 ERROR            PET001 med_phases_aofluxes_mod.F90:292 Invalid argument  - Passing error in return code
20211230 134551.548 ERROR            PET001 MED:src/addon/NUOPC/src/NUOPC_ModelBase.F90:2418 Invalid argument  - Passing error in return code
20211230 134551.548 ERROR            PET001 EARTH Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:3353 Invalid argument  - Phase 'med_phases_aofluxes_run' Run for modelComp 1 did not return ESMF_SUCCESS
20211230 134551.548 ERROR            PET001 EARTH Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:3619 Invalid argument  - Passing error in return code
20211230 134551.548 ERROR            PET001 EARTH Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:3268 Invalid argument  - Passing error in return code
20211230 134551.548 ERROR            PET001 MAIN_NEMS.F90:446 Invalid argument  - Aborting NEMS
20211230 134551.548 INFO             PET001 Finalizing ESMF

uturuncoglu · 2021-12-30T20:50:23Z

@mvertens I think I need to update ESMF version. Right? Which version CESM is using currently?

uturuncoglu · 2021-12-30T20:53:53Z

@mvertens okay. I found it. Compiling with newer version. I hope the UFS model is compatible with newer version of ESMF.

uturuncoglu · 2021-12-30T21:11:47Z

@mvertens Okay. I update ESMF to the same version used under CESM and run the model again. Now I am getting following error,

20211230 140923.374 ERROR            PET002 Destination id=2544 NOT found in weight matrix.
20211230 140923.375 ERROR            PET002 ESMCI_Mesh_Regrid_Glue.C:335 ESMCI_regrid_create() Arguments are incompatible  - - There exist destination points (e.g. id=2544) which can't be mapped to any source cell
20211230 140923.375 ERROR            PET002 ESMCI_Mesh_Regrid_Glue.C:558 ESMCI_regrid_create() Arguments are incompatible  - Internal subroutine call returned Error
20211230 140923.375 ERROR            PET002 ESMCI_MeshCap.C:1599 MeshCap::regrid_create() Arguments are incompatible  - Internal subroutine call returned Error
20211230 140923.375 ERROR            PET002 ESMF_Regrid.F90:372 ESMF_RegridStore Arguments are incompatible  - Internal subroutine call returned Error
20211230 140923.375 ERROR            PET002 ESMF_FieldRegrid.F90:1204 ESMF_FieldRegridStoreNX Arguments are incompatible  - Internal subroutine call returned Error
20211230 140923.375 ERROR            PET002 med_phases_aofluxes_mod.F90:845 Arguments are incompatible  - Passing error in return code
20211230 140923.375 ERROR            PET002 med_phases_aofluxes_mod.F90:423 Arguments are incompatible  - Passing error in return code
20211230 140923.375 ERROR            PET002 med_phases_aofluxes_mod.F90:292 Arguments are incompatible  - Passing error in return code
20211230 140923.375 ERROR            PET002 MED:src/addon/NUOPC/src/NUOPC_ModelBase.F90:2215 Arguments are incompatible  - Passing error in return code
20211230 140923.375 ERROR            PET002 EARTH Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:3343 Arguments are incompatible  - Phase 'med_phases_aofluxes_run' Run for modelComp 1 did not return ESMF_SUCCESS
20211230 140923.375 ERROR            PET002 EARTH Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:3609 Arguments are incompatible  - Passing error in return code
20211230 140923.375 ERROR            PET002 EARTH Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:3258 Arguments are incompatible  - Passing error in return code
20211230 140923.375 ERROR            PET002 MAIN_NEMS.F90:446 Arguments are incompatible  - Aborting NEMS
20211230 140923.375 INFO             PET002 Finalizing ESMF

I'll try to find the issue but we might need input from Bob. It could be related with the FV3 grid but I am not sure.

uturuncoglu · 2021-12-30T21:16:29Z

It seems following call causes error in my case,

    call ESMF_FieldRegridStore(field_a, field_x, routehandle=rh_agrid2xgrid_bilinr, &
         regridmethod=ESMF_REGRIDMETHOD_BILINEAR, dstMaskValues=(/0/), rc=rc)
    if (chkerr(rc,__LINE__,u_FILE_u)) return

Since this is recently added there could be bug in there. @oehmke @rsdunlapiv do you have any input or suggestion about it. The full trace can be seen in the previous comment.

mvertens · 2021-12-31T18:26:42Z

@uturuncoglu - since this works for me can we put an if clause for the model so that we can use this for cesm?

uturuncoglu · 2022-01-01T03:07:26Z

@mvertens if you would like to go with temporal solution that is fine for me and I could test it again when you update the branch. It seems that the issue is in ESMF side but I am not full sure.

mvertens · 2022-01-02T17:51:03Z

@uturuncoglu - I have updated the code to only use the agrid->xgrid bilinear mapping if the model is not hafs. Could you please try this again with your case. Also - could you please raise an issue regarding the problem you found - if this new code now passes for you.

uturuncoglu · 2022-01-02T19:48:56Z

@mvertens application is not hafs. It is nems_frac_aoflux but this is not available to you since it is in my fork.

mvertens · 2022-01-02T23:02:23Z

@uturuncoglu - how do you want to proceed with this PR? I can set the model to nems_frac_aoflux for now - and it will simply skip that code unless you have merged this into your branch? Does that sound reasonable?

uturuncoglu · 2022-01-03T17:06:19Z

@mvertens I think you could revert your recent changes back. Then, I will update my fork to put your logic. Once my fork merged with the CMEPS master, we will have the fix. In the meantime, I could bring this issue in the next ESMF core call to discuss possible solutions in the ESMF side. Let me know what do you think?

mvertens · 2022-01-03T17:28:28Z

@uturuncoglu - okay - that is fine with me. I'll do that and push things back. Are you then okay with accepting this PR?
Can you also raise an issue so that we are aware of this?

uturuncoglu · 2022-01-03T17:29:52Z

@mvertens I also confirm that patch gives similar error,

20220103 102309.917 ERROR            PET079 Destination id=185119 NOT found in weight matrix.
20220103 102309.917 ERROR            PET079 ESMCI_Mesh_Regrid_Glue.C:335 ESMCI_regrid_create() Arguments are incompatible  - - There exist destination points (e.g. id=185119) which can't be mapped to any source cell
20220103 102309.917 ERROR            PET079 ESMCI_Mesh_Regrid_Glue.C:558 ESMCI_regrid_create() Arguments are incompatible  - Internal subroutine call returned Error
20220103 102309.917 ERROR            PET079 ESMCI_MeshCap.C:1599 MeshCap::regrid_create() Arguments are incompatible  - Internal subroutine call returned Error
20220103 102309.917 ERROR            PET079 ESMF_Regrid.F90:372 ESMF_RegridStore Arguments are incompatible  - Internal subroutine call returned Error
20220103 102309.917 ERROR            PET079 ESMF_FieldRegrid.F90:1204 ESMF_FieldRegridStoreNX Arguments are incompatible  - Internal subroutine call returned Error
20220103 102309.917 ERROR            PET079 med_phases_aofluxes_mod.F90:848 Arguments are incompatible  - Passing error in return code
20220103 102309.917 ERROR            PET079 med_phases_aofluxes_mod.F90:423 Arguments are incompatible  - Passing error in return code
20220103 102309.917 ERROR            PET079 med_phases_aofluxes_mod.F90:292 Arguments are incompatible  - Passing error in return code
20220103 102309.917 ERROR            PET079 MED:src/addon/NUOPC/src/NUOPC_ModelBase.F90:2215 Arguments are incompatible  - Passing error in return code
20220103 102309.917 ERROR            PET079 EARTH Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:3343 Arguments are incompatible  - Phase 'med_phases_aofluxes_run' Run for modelComp 1 did not return ESMF_SUCCESS
20220103 102309.917 ERROR            PET079 EARTH Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:3609 Arguments are incompatible  - Passing error in return code
20220103 102309.917 ERROR            PET079 EARTH Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:3258 Arguments are incompatible  - Passing error in return code
20220103 102309.917 ERROR            PET079 MAIN_NEMS.F90:446 Arguments are incompatible  - Aborting NEMS
20220103 102309.917 INFO             PET079 Finalizing ESMF

I am currently testing with 2nd order conservative. I'll let you know soon about the results.

uturuncoglu · 2022-01-03T18:00:22Z

@mvertens if I replaces patch and bilinear with 2nd order, it works. I have still minor differences when I compare it with the older version of CMEPS. Are those expected differences. I know you update the normalization but I was expecting to have differences around icy regions but after one day simulation, I have difference also over open ocean. I could investigate further but testing again to be sure but I don't want to delay the PR. Let me know what do you think?

mvertens · 2022-01-03T18:04:46Z

@uturuncoglu - if you are referring to xgrid only - I'm not sure. Were you using first order conservative before?

uturuncoglu · 2022-01-03T18:05:41Z

@mvertens let me check the version that I was using perviously. maybe you are right and I was using 1st order.

uturuncoglu · 2022-01-03T18:09:01Z

@mvertens I think it was using 2nd order for u and v and 1st order for the rest (I think this is default for creating route handle with xgrid, right?). Let me try same with new version of the code and maybe the difference will disappear.

uturuncoglu

@mvertens I tested 2nd order with u and v and 1st order for the other variables but I have still differences. Maybe it is introduced by the normalization. It is very minimal (around 1.0e-12) in the first time step. Anyway, I am approving this PR at this point and I'll open an issue related with the patch and bilinear interpolations.

uturuncoglu · 2021-12-27T16:25:32Z

+    call ESMF_FieldRegridStore(xgrid, field_x, field_o, routehandle=rh_xgrid2ogrid, rc=rc)
    if (chkerr(rc,__LINE__,u_FILE_u)) return
-    ! call ESMF_FieldRegridStore(xgrid, lfield_o, lfield_x, routehandle=rh_ogrid2xgrid_2ndord, &
+    ! call ESMF_FieldRegridStore(xgrid, field_o, field_x, routehandle=rh_ogrid2xgrid_2ndord, &


This might be removed from the code but if you are planing to activate it later, that is fine.

uturuncoglu · 2022-01-03T20:49:28Z

@mvertens I created issue #264 to track it.

rsdunlapiv · 2022-01-18T16:53:02Z

@mvertens just checking on the timeline for getting this merged in?

mvertens · 2022-01-18T17:38:52Z

@uturuncoglu - can I go ahead and merge this?

uturuncoglu · 2022-01-18T17:54:42Z

@mvertens I created an issue in ESMF side about using bilinear and patch under UFS. Bob is looking for the issue. I think you could merge this and I could put workaround for it in the UFS side. Thanks for asking.

mvertens · 2022-01-18T19:35:07Z

Thank you!

…

On Tue, Jan 18, 2022 at 10:54 AM Ufuk Turunçoğlu ***@***.***> wrote: @mvertens <https://github.com/mvertens> I created an issue in ESMF side about using bilinear and patch under UFS. Bob is looking for the issue. I think you could merge this and I could put workaround for it in the UFS side. Thanks for asking. — Reply to this email directly, view it on GitHub <#262 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AB4XCE7XKUK2JUXLT7H5CS3UWWSO7ANCNFSM5KXH4CYQ> . Triage notifications on the go with GitHub Mobile for iOS <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675> or Android <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>. You are receiving this because you were mentioned.Message ID: ***@***.***>

-- Dr. Mariana Vertenstein CESM Software Engineering Group Head National Center for Atmospheric Research Boulder, Colorado Office 303-497-1349 Email: ***@***.***

Plumber updates This addresses ESCOMP#248 In order to implement PLUMBER capabilities

changes to fix budget issues for aoflux on xgrid

b2979b5

mvertens requested review from DeniseWorthen, jedwards4b and uturuncoglu December 25, 2021 00:44

jedwards4b approved these changes Dec 26, 2021

View reviewed changes

DeniseWorthen approved these changes Dec 27, 2021

View reviewed changes

only use binlinear agrid->xgrid mapping if not hafs

2b333b6

uturuncoglu approved these changes Jan 3, 2022

View reviewed changes

uturuncoglu mentioned this pull request Jan 3, 2022

Issue with patch and bilinear with xgrid under UFS #264

Open

backed out hafs coupling mode changes for agrid->xgrid

39a94a9

mvertens merged commit 4322867 into ESCOMP:master Jan 20, 2022

mvertens deleted the feature/xgrid_updates branch June 16, 2023 08:06

korsbakken pushed a commit to korsbakken/CMEPS that referenced this pull request Oct 23, 2025

Merge pull request ESCOMP#262 from TeaganKing/plumber

f199f9f

Plumber updates This addresses ESCOMP#248 In order to implement PLUMBER capabilities

Conversation

mvertens commented Dec 25, 2021 • edited by DeniseWorthen Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description of changes

Specific notes

Uh oh!

uturuncoglu commented Dec 27, 2021

Uh oh!

mvertens commented Dec 27, 2021

Uh oh!

DeniseWorthen left a comment

Choose a reason for hiding this comment

Uh oh!

uturuncoglu commented Dec 28, 2021

Uh oh!

mvertens commented Dec 28, 2021

Uh oh!

uturuncoglu commented Dec 30, 2021

Uh oh!

mvertens commented Dec 30, 2021 via email • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

uturuncoglu commented Dec 30, 2021

Uh oh!

uturuncoglu commented Dec 30, 2021

Uh oh!

uturuncoglu commented Dec 30, 2021

Uh oh!

uturuncoglu commented Dec 30, 2021

Uh oh!

uturuncoglu commented Dec 30, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

uturuncoglu commented Dec 30, 2021

Uh oh!

mvertens commented Dec 31, 2021 via email • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

uturuncoglu commented Jan 1, 2022

Uh oh!

mvertens commented Jan 2, 2022

Uh oh!

uturuncoglu commented Jan 2, 2022

Uh oh!

mvertens commented Jan 2, 2022

Uh oh!

uturuncoglu commented Jan 3, 2022 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mvertens commented Jan 3, 2022

Uh oh!

uturuncoglu commented Jan 3, 2022

Uh oh!

uturuncoglu commented Jan 3, 2022

Uh oh!

mvertens commented Jan 3, 2022

Uh oh!

uturuncoglu commented Jan 3, 2022

Uh oh!

uturuncoglu commented Jan 3, 2022

Uh oh!

uturuncoglu left a comment

Choose a reason for hiding this comment

Uh oh!

uturuncoglu Dec 27, 2021

Choose a reason for hiding this comment

Uh oh!

uturuncoglu commented Jan 3, 2022

Uh oh!

rsdunlapiv commented Jan 18, 2022

Uh oh!

mvertens commented Jan 18, 2022

Uh oh!

uturuncoglu commented Jan 18, 2022

Uh oh!

mvertens commented Jan 18, 2022 via email

Uh oh!

Reviewers

Assignees

Labels

mvertens commented Dec 25, 2021 •

edited by DeniseWorthen

Loading

mvertens commented Dec 30, 2021 via email •

edited

Loading

uturuncoglu commented Dec 30, 2021 •

edited

Loading

mvertens commented Dec 31, 2021 via email •

edited

Loading

uturuncoglu commented Jan 3, 2022 •

edited

Loading