Skip to content

Unified ugwp#55

Merged
DomHeinzeller merged 19 commits into
NOAA-GSL:gsd/developfrom
mdtoy:unified_ugwp
Nov 17, 2020
Merged

Unified ugwp#55
DomHeinzeller merged 19 commits into
NOAA-GSL:gsd/developfrom
mdtoy:unified_ugwp

Conversation

@mdtoy
Copy link
Copy Markdown

@mdtoy mdtoy commented Oct 13, 2020

Hello,

This PR introduces the "unified_ugwp" gravity wave drag scheme, which combines three drag schemes: 1) the "cires_ugwp" scheme, 2) the GSL drag suite "drag_suite", and 3) the new "ugwp_v1" scheme developed by Valery Yudin of CIRES. The first two schemes can be regression-tested against their original respective schemes, while the "ugwp_v1" is still under development. To activate the various options, new namelist options are required, which are available from the developers.

Regards,
Mike Toy

@mdtoy mdtoy requested a review from DomHeinzeller as a code owner October 13, 2020 16:21
Copy link
Copy Markdown

@DomHeinzeller DomHeinzeller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a beautiful, clean implementation of the combined gravity wave drag schemes. I was very picky along the way to request passing constants used deep inside the physics from the host model, and to reduce clutter to stdout/stderr if possible. Plus a few other, small changes.

Here is my suggestion to address the constants and stdout/stderr concerns:

Constants

It seems everything draws constants from module ugwp_common_v1. If we could find a way to set those constants as part of/first task of the init call, then we can leave all code unchanged and still get the host model constants.

This will most likely change the answer, so we should make this change in a controlled way: first run the tests to verify that all existing regression tests pass when using internal constants. Then swap to use host model constants (maybe even using #ifdef statements in ugwp_common_v1 and make sure by manual inspection that the results are still acceptable.

stdout/stderr

There is a large amount of output written to stdout and stderr in the current code. This may be usedul for development, testing and debugging. I would wrap all but the absolutely necessary print statements in an if (debug) block and then have a single debug variable, maybe in ugwp_common_v1, that can be set to .true. by the user (or automatically using

#ifdef DEBUG
logical, parameter :: debug = .true.
#else
logical, parameter :: debug = .false.
#endif 

Either way, the compiler knows at compile time what the value is and can optimize out all these blocks if no needed.

Lastly, I would suggest to remove any existing cires_ugwp files (CCPP entry points and dependencies) that are no longer needed.

Comment thread physics/cires_orowam2017.F90 Outdated
tau_kx(:) = tau_kx(:)/tau_norm

if (kdt == 1) then
771 format( 'vay-oro19 ', 3(2x,F8.3))
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We would like to reduce the amount of clutter written to stdout, but for the time being and while this is under development, it is ok for me.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay. Leaving it as-is for now.

Comment thread physics/cires_orowam2017.F90 Outdated
! arad => con_rerth
implicit none

real, parameter :: grav =9.81, cpd = 1004.
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Constants should come from the host model through the argument list, if possible. Most if not all of these are defined in fv3atm already.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Leaving as-is for now, but am eliminating "use" statements which call for the physical constants.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

IMO, it is dangerous for every scheme to be defining their own physical constants. They should be defined in one place and used across physics. In the CCPP, sending constants through the argument list enables this -- each host model that uses the CCPP defines physical constants in their own way (which is why you can't rely on "use physcons") and as long as the host provides metadata for the constants so that the CCPP knows what to pass in, it all works as expected with common values amongst physics. In practice, for FV3, the constants may be defined in physcons.F90, but they have metadata and this metadata is made known to the CCPP framework at build time.

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I am still seeing places in the code that do import constants from module ugwp_common_v1? Did I misread your reply?

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I see what you mean, but for the subroutines that are actually called in the ugwp_v1 version, the physical constants that are used are passed in through the argument lists. I would like to leave the other subroutines as-is, as I believe a lot of them will be removed after cleaning up (with Valery next month).

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for the explanation. As long as you remember all these details ... (I won't, that's for sure).

Comment thread physics/cires_ugwp_initialize_v1.F90 Outdated
Comment thread physics/cires_ugwp_triggers_v1.F90 Outdated
Comment thread physics/cires_vert_orodis_v1.F90 Outdated
Comment thread physics/cires_vert_orodis_v1.F90 Outdated
Comment thread physics/cires_vert_orodis_v1.F90 Outdated
Comment thread physics/unified_ugwp_post.F90 Outdated
@mdtoy
Copy link
Copy Markdown
Author

mdtoy commented Oct 20, 2020 via email

@DomHeinzeller
Copy link
Copy Markdown

DomHeinzeller commented Oct 20, 2020 via email

@climbfuji
Copy link
Copy Markdown

@mdtoy I can't follow when you think the PR is ready again. Can you please let me know? If you could resolve the comments that you addressed, leave the others open and create an issue in this repository to remind ourselves about them, that would be great.

real, parameter :: gr2 = grav*gor
real, parameter :: grcp = grav*rcpd, gocp = grcp
real, parameter :: rcpdl = cpd*rgrav ! 1/[g/cp] == cp/g
real, parameter :: grav2cpd = grav*grcp ! g*(g/cp)= g^2/cp
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Even some of these derived constants have CCPP standard names, BTW. Where they don't, and they must be calculated locally, they will of course need to no longer be defined as parameters.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I had found that the derived ratio "fv" had the CCPP standard name "con_fvirt", and I used it in my previous revision from last week, where I did recalculate them locally (not as parameters). For now I would like to leave ugwp_common_v1 as-is even though I'm not using any physical constants from it. We can get rid of unused constants for our final ugwp_v1 version. I hope that's okay.

I just found some more ratios in ccpp/physics/physics/physcons.F90 and I could pass them through GFS_typedefs.meta later if necessary. Is physcons.F90 correct source? Also, are we free to add our own ratios to physcons.F90? (BTW, why aren't more of the ratios already in GFS_typedefs.meta?)

Comment thread physics/cires_vert_orodis_v1.F90 Outdated
Comment thread physics/cires_vert_orodis_v1.F90 Outdated
Comment thread physics/ugwp_driver_v0.F
!-----------------------------------------------------------
use machine, only : kind_phys
use physcons, only : con_cp, con_g, con_rd, con_rv
use physcons, only : con_cp, con_g, con_rd, con_rv, &
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Same comment about physical constants.

Copy link
Copy Markdown
Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This section of code is carried over from cires_ugwp_v0, already in the gsd/develop branch. The unified_ugwp (v1) code doesn't call this, so I'm wondering if we can keep it as-is.

Comment thread physics/unified_ugwp.meta
Copy link
Copy Markdown

@grantfirl grantfirl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As someone outside of GSD, I should have zero influence on what goes on in this fork, but I was asked to review. I'm honestly very confused about the gravity wave drag code unification strategy. What purpose does it serve to push all of these schemes into one CCPP scheme? From a CCPP point-of-view, and a GWD layman, if there are differing algorithms for solving the same parameterization problem, why should they not be distinct CCPP schemes? If the goal is to coalesce on the best algorithm, why are multiple versions of an algorithm being kept around under the veil of one "unified" one? It could certainly be that I'm just not understanding the code, but if there are going to be a bunch of namelist options to use one of Valery's versions or the GSD drag suite or the older GFS scheme, I think that there needs to be REALLY good documentation (with flow diagrams) that help a user understand what is actually being executed with the various namelist options. I'm not seeing that right now. Also, does anyone else find it ironic that the word "unified" appears twice in the name "unified UGWP"?

@climbfuji
Copy link
Copy Markdown

As someone outside of GSD, I should have zero influence on what goes on in this fork, but I was asked to review. I'm honestly very confused about the gravity wave drag code unification strategy. What purpose does it serve to push all of these schemes into one CCPP scheme? From a CCPP point-of-view, and a GWD layman, if there are differing algorithms for solving the same parameterization problem, why should they not be distinct CCPP schemes? If the goal is to coalesce on the best algorithm, why are multiple versions of an algorithm being kept around under the veil of one "unified" one? It could certainly be that I'm just not understanding the code, but if there are going to be a bunch of namelist options to use one of Valery's versions or the GSD drag suite or the older GFS scheme, I think that there needs to be REALLY good documentation (with flow diagrams) that help a user understand what is actually being executed with the various namelist options. I'm not seeing that right now. Also, does anyone else find it ironic that the word "unified" appears twice in the name "unified UGWP"?

Thanks, @grantfirl for taking the time to review the code and flagging the constants/stop issues that I missed in my first round of reviews. Regarding your general concern. My understanding was, but @mdtoy please correct me if I am wrong, that the goal of putting those various versions under one hood is to be able to experiment with them and agree on what is the best solution for implementation in GFSv17 (?). Once this has been decided, the code would be cleaned up and only what is needed will be left in the final, unified version (maybe with a triple "unified" in the name then???).

@mdtoy
Copy link
Copy Markdown
Author

mdtoy commented Oct 23, 2020

That's correct Dom, this is a transitional scheme for testing the various NOAA drag schemes to select and tune the best combination. For the final scheme, we'll find a better name than unified^nth power.

@mdtoy
Copy link
Copy Markdown
Author

mdtoy commented Oct 26, 2020

To follow up on my previous comment, the unified_ugwp scheme doesn't just put three separate schemes into one for calling each one individually (which is an option, of course), but instead different pieces of the schemes can be combined with the others to come up with new possibilities (e.g., ugwp_v0 GWD+blocking combined with GSL small-scaleGWD + turbulent orographic form drag). This is the testing framework which will help with deciding on the final combination.

@mdtoy
Copy link
Copy Markdown
Author

mdtoy commented Oct 26, 2020

I pushed my changes for your review.

@mdtoy
Copy link
Copy Markdown
Author

mdtoy commented Oct 27, 2020

Before notifying that my PR changes were ready for review, I had forgotten to test if the new code would compile. There was a slight bug, which was fixed and pushed just now. The bug was in cires_vert_orodis_v1.F90.

Copy link
Copy Markdown

@DomHeinzeller DomHeinzeller left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

What kind of testing have you done with this code after addressing the reviewer comments? Did you run (some of) the rt.sh regression tests? Trying to figure out what tests I have to run. Thanks ...

@mdtoy
Copy link
Copy Markdown
Author

mdtoy commented Oct 27, 2020

I did an RT after your suggested changes last week, but I haven't done another since yesterday's changes suggested by Grant. If you like, I can perform them this evening.
I'm using this shortened testing script:
/scratch1/BMC/wrfruc/mtoy/git_local/ufs-weather-model/tests/rt_ccpp_gsd_short.conf

The fv3_ccpp_gsd_drag_suite fails because I haven't put in the required input.nml changes automatically. I redo it manually and it checks out. Let me know if you want me to create scripts that automate the tests, including three more tests that I do with the unified_ugwp scheme to test for reproducibility of the three schemes.

@climbfuji
Copy link
Copy Markdown

I did an RT after your suggested changes last week, but I haven't done another since yesterday's changes suggested by Grant. If you like, I can perform them this evening.
I'm using this shortened testing script:
/scratch1/BMC/wrfruc/mtoy/git_local/ufs-weather-model/tests/rt_ccpp_gsd_short.conf

The fv3_ccpp_gsd_drag_suite fails because I haven't put in the required input.nml changes automatically. I redo it manually and it checks out. Let me know if you want me to create scripts that automate the tests, including three more tests that I do with the unified_ugwp scheme to test for reproducibility of the three schemes.

That would be great, thank you!

@mdtoy
Copy link
Copy Markdown
Author

mdtoy commented Oct 29, 2020

I performed regression tests of the unified_ugwp scheme, which consisted of the following 6 tests using the test script tests/rt_ccpp_gsd_unified_ugwp.conf (they all passed):

  1. CCPP_SUITE = FV3_GSD_v0
  2. CCPP_SUITE = FV3_GSD_v0_drag_suite
  3. CCPP_SUITE = FV3_GSD_v0_no_drag
  4. CCPP_SUITE = FV3_GSD_v0_unified_ugwp_suite, input.nml settings to reproduce FV3_GSD_v0 scheme
  5. CCPP_SUITE = FV3_GSD_v0_unified_ugwp_suite, input.nml settings to reproduce FV3_GSD_v0_drag_suite scheme
  6. CCPP_SUITE = FV3_GSD_v0_unified_ugwp_suite, input.nml settings to reproduce FV3_GSD_v0_no_drag scheme
    See the following spreadsheet for a more visual explanation:
    https://docs.google.com/spreadsheets/d/13mEhzSF7bZrw8fHJU_LmHBAHvFkiTn4PZ395vvzJSa4/edit#gid=0

The baselines were created with the test script:
tests/rt_ccpp_gsd_short.conf (baseline directories for the unified_ugwp
These baselines were created with ccpp-physics code from the gsd/develop branch, using three suites: FV3_GSD_v0, FV3_GSD_v0_drag_suite, and FV3_GSD_v0_no_drag (note that the "no_drag" scheme doesn't need to be merged with the authoritative branch -- it is just to test the ability for the unified scheme to reproduce the no-drag results with all the scheme selection namelist variables set to .false.). The "baselines" for the unified_ugwp code were created manually by symlinks to the three relevant directories in the REGRESSION_TEST_INTEL directory.

Feel free to modify/delete my test scripts as needed.

Thanks.

@DomHeinzeller
Copy link
Copy Markdown

@mdtoy, can you please make sure that your ccpp-physics PR is up to date, i.e. includes everything from NOAA-GSD gsd/develop ? I don't know if it is, just checking.

In ccpp/physics, check that one of your remotes points to https://github.com/NOAA-GSD/ccpp-physics (assume the remote is named as upstream for the moment, and that your fork is named as origin). Then:

git remote update
git pull upstream gsd/develop
git push origin unified_ugwp

Then go back to FV3 and see if git status shows new commits for ccpp/physics. If it does, do

git add ccpp/physics
git commit -m "Update submodule pointer for ccpp-physics"
git push origin unified_ugwp

@mdtoy
Copy link
Copy Markdown
Author

mdtoy commented Nov 5, 2020

I just merged latest NOAA-GSD gsd/develop and pushed to my fork (PR).

@DomHeinzeller
Copy link
Copy Markdown

@mdtoy I am getting one unexpected regression test failure when running the full regression tests (rt.conf) on hera.intel against the official baseline: fv3_ccpp_gfdlmprad_gwd - this test crashes:

  0:  af create fcst fieldbundle, name=dynrc=           0
  0:  af create fcst fieldbundle, name=phy_nearest_stodrc=           0
  0:  af create fcst fieldbundle, name=phy_bilinearrc=           0
  0:  in fcst,init total time:    1.80379605293274
117: forrtl: severe (174): SIGSEGV, segmentation fault occurred
117: Image              PC                Routine            Line        Source
117: fv3.exe            000000000372A4BC  Unknown               Unknown  Unknown
117: libpthread-2.17.s  00002BA3C3F9E630  Unknown               Unknown  Unknown
117: fv3.exe            0000000002A10891  Unknown               Unknown  Unknown
117: fv3.exe            0000000002891DDD  Unknown               Unknown  Unknown
117: fv3.exe            0000000002794CA4  Unknown               Unknown  Unknown
117: fv3.exe            000000000251BE4A  Unknown               Unknown  Unknown
117: fv3.exe            0000000001B152EB  Unknown               Unknown  Unknown
117: libiomp5.so        00002BA3C23A7A43  __kmp_invoke_micr     Unknown  Unknown
117: libiomp5.so        00002BA3C236B2C6  __kmp_fork_call       Unknown  Unknown
117: libiomp5.so        00002BA3C232ABB0  __kmpc_fork_call      Unknown  Unknown
117: fv3.exe            0000000001B14CB1  Unknown               Unknown  Unknown
117: fv3.exe            0000000001AE0ABB  Unknown               Unknown  Unknown
117: fv3.exe            0000000001AD4753  Unknown               Unknown  Unknown
117: fv3.exe            0000000000A3696E  _ZN5ESMCI6FTable1        2009  ESMCI_FTable.C
117: fv3.exe            0000000000A3A5B6  ESMCI_FTableCallE         747  ESMCI_FTable.C

I will need to run this test in DEBUG mode to see where it crashes. So far, the other tests are looking good, but not all votes have been counted yet ;-)

@mdtoy
Copy link
Copy Markdown
Author

mdtoy commented Nov 9, 2020

Thanks for the update. Let me know what the results of the debug-mode test are.

@DomHeinzeller
Copy link
Copy Markdown

This PR has been pulled into #61 and will be merged automatically. Do not merge manually.

@DomHeinzeller DomHeinzeller added the do not merge Something is wrong, do not merge label Nov 17, 2020
DomHeinzeller added a commit that referenced this pull request Nov 17, 2020
@DomHeinzeller DomHeinzeller merged commit 52ec168 into NOAA-GSL:gsd/develop Nov 17, 2020
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

do not merge Something is wrong, do not merge

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants