Add FV3 suite WoFS_v0#1158
Conversation
This reverts commit 25a09b6.
|
@junwang-noaa Only one of the drafts will be used, depending on whether the RT tests for v1nssl should be converted to WoFS_v0 or not. I may have jumped the gun a little on that.. 1158 just adds the new SDF without touching the RT files, which is the easiest path. |
|
@MicroTed could you please try to run case with this new SDF by reversing MPI layout in input.nml file. For example, if you have layout = 6,11 - try with layout = 11,6 (this will change MPI decomposition) NOTE Our current SDF (suite_FV3_GFS_v15_thompson_mynn_lam3km.xml) is reproducing results only with coarse resolution (currently tested in UFS regression tests), but at 3km it is creating different results. |
Sure thing. Is there any particular current RT test that would be good?
My experience with changing the layout is that optimization has to be off completely (to turn off vectorization). Otherwise there can be a change in the "leftover" operations that don't fit into the vector and have different round-off characteristics. It has been a few years since I've tested that, so YMMV. |
|
@MicroTed There are two tests in rt,conf: First one uses low resolution and second one high resolution. Just reminder to use -DREPRO=ON in COMPILE line, and of course, use suite_FV3_WoFS_v0.xml in same line. |
|
@RatkoVasic-NOAA |
Right |
|
@MicroTed May I ask if you ran the decomposition test as Ratko suggested (reversing MPI layout in input.nml file)? We need to run the ORT test(threading, decomposition, restart reproducibility test and debug test ) for new physics suite. Thanks. |
Not completely column-only -- the Thompson interface (mp_thompson.F90) has 2D loops that will vary in width with MPI. The NSSL scheme is 2D within the microphysics code (works on 2D slabs rather than just columns) @junwang-noaa I did try flipping the layout on the regional_3km, and results do change, even after changing the REPRO options to '-fp-model precise' |
|
@MicroTed is the test reproducing with restart runs? |
|
Yes. We want it to be in the SRW v2 release. Could you provide more details about the decomposition reproducing? Actually, this PR just adds one CCPP suite file. I am wondering why it will impact the decomposition? |
|
@ywangwof The requirement for public release is that the suite files are well tested. When developers run the test using the suite file (e.g. regional_3km_wofs) with different threads, different decomposition, and restart they will get same answer. The test can also run in debug mode. These tests are required for public release (@ligiabernardet please let me know if the requirement is changed). |
|
In the |
@MinsukJi-NOAA Thanks for pointing that out! I renamed the file. |
@RatkoVasic-NOAA At this point, it seems reasonable to note that there is an issue somewhere and move on? |
|
Pinging @junwang-noaa, since I think Ratko may be on leave. It sounds like there is a systematic problem with physics reproducibility at higher resolutions that we'll need to address at some point, but it's unrelated to the SDFs themselves. Therefore, do you think we could get this PR back into the merge queue? Thanks! |
|
The dcp test for the global works if nssl_invertccn=.false. I don't understand yet why the "true" option fails, but it is used for LAM domains to avoid needing BCs for the CCN variable (BC can be just zero). |
|
The dcp issue seems to be fixed. I opened a PR for CCPP for both the restart and dcp fixes: |
Passing of the |
|
@MicroTed I have made a PR to your |
@MinsukJi-NOAA OK, good to know, and thanks for doing that -- I have't gone back to the 3km test yet. Considering that FV3_GFS_v15_thompson_mynn_lam3km also has trouble with decomposition for the 3km test, maybe it is not surprising. Maybe even expected? |
Modify opnReqTest related scripts to test regional_3km_wofs
|
@MicroTed If I understand correctly, the suite_FV3_RRFS_V1nssl.xml has decomposition test passed. So for the suite file suite_FV3_WoFS_v0.xml, are you considering using noahmpdrv instead of lsm_noah for the regional_3km_wofs decomposition test? |
Yes, it passes the rrfs_v1nssl RT case, which is a global coarse grid. But, like the lam3km suite, apparently not on the 3km regional grid. Is there a regional case that passes decomposition? The rt.conf has a comment that rap_decomp and rap_sfcdiff_decomp also don't work. Is there one that does work? |
|
If the suite_FV3_WoFS_v0.xml is going to be used by SRW as a regional test, I think the decomposition test needs to be fixed, if not please set up a global test case with suite_FV3_WoFS_v0.xml, so the ORT will pass. |
Sure, I can add the global test. It works fine. I'm also testing regional_control adapted for the WoFS_v0 suite, but that has a note that restart tests don't work? Update: "Test dcp regional_wofs PASS" So whatever the problem is with the 3km test, it is likely that same thing that affects both the lam3km and wofs suites. |
|
@MicroTed I'd suggest to add the test that will be used in public release and fix the issue if there is. I think Ratko is also debugging the lam3km test case, maybe you can work with him. |
|
Update: We found that the regional_3km_wofs passes the decomposition test if nrows_blend=0. The problem with nrows_blend > 0 appears to be the setting of loop limits for the blending points in the corner MPI tiles (in the bc_time_interpolation subroutine in fv_regional_bc.F90). |
Nevermind this -- I think there is another route.Would be OK to proceed with the FV3 PR to add the WoFS suite so it can be included in the SRW-v3? NOAA-EMC/ufsatm#514 |
|
A PR with the suite files was committed to the SRW public release branch. |
|
This PR included changes to GFDL_atmos_cubed_sphere which have already been merged. |
|
Closing in favor of PR #1460 |
12 participants
PR Checklist
This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR. Please consult the ufs-weather-model wiki if you are unsure how to do this.
This PR has been tested using a branch which is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR
An Issue describing the work contained in this PR has been created either in the subcomponent(s) or in the ufs-weather-model. The Issue should be created in the repository that is most relevant to the changes in contained in the PR. The Issue and the dependent sub-component PR
are specified below.
Results for one or more of the regression tests change and the reasons for the changes are understood and explained below.
New or updated input data is required by this PR. If checked, please work with the code managers to update input data sets on all platforms.
Instructions: All subsequent sections of text should be filled in as appropriate.
Description
Adds new suite definition file to FV3 (FV3_WoFS_v0) to use the NSSL cloud microphysics scheme with Noah LSM. For use use with the next SRW app release.
Testing
Testing done on Jet
Dependencies
If testing this branch requires non-default branches in other repositories, list them. Those branches should have matching names (ideally).
Do PRs in upstream repositories need to be merged first?
"waiting for other repos"