Bring in CDEPS, CMEPS/CDEPS component level PIO initialization and add HAFS specific RTs by uturuncoglu · Pull Request #8 · hafs-community/ufs-weather-model

uturuncoglu · 2021-03-10T04:39:12Z

Description

This PR includes;

Bring in CDEPS to HADS model
Update PIO initialization and move it from NEMS to component level PIO
Add HAFS specific following RTs (FV3 is low-resolution - 13 km)
- Standalone FV3 for HAT10 - fv3_ccpp_regional_natl_c192
- Fully coupled FV3+HYCOM - cpld_regional_natl_c192
- FV3+DOCN (regional, MOM6 HAT 10 domain) - cpld_regional_natl_c192_docn_mom6
- FV3+DOCN (global, OISST) - cpld_regional_natl_c192_docn_oisst
- DATM (global, ERA5)+HYCOM - datm_era5_hycom
- I also removed all data (datm and docn) RT under rt_hafs.conf since this is not testing the system complete and it was a intermediate step to test in the lack of RTs.

It also requires following PR to be merged,

The input files are baselines for newly defined RTs can be found in /work/noaa/stmp/tufuk/stmp/tufuk/FV3_RT/REGRESSION_TEST_INTEL on Orion and eventually these need to go to the official location. All new RTs are both defined in rt.conf and rt_hafs.conf.

What bug does it fix, or what feature does it add?
It adds capability to use CDEPS and CMEPS together and brings component level PIO support for those components.

Is a change of answers expected from this PR?
No. All RT passed without any issue.

Are any library updates included in this PR (modulefiles etc.)?
No. To use pnetcdf support along with PIO in CMEPS and CDEPS, PIO needs to be installed (along with HPC-stack) with pnetcdf support. The latest version on PIO (5.2.3) that includes fix for pnetcdf support can be found in https://github.com/NCAR/ParallelIO/releases/tag/pio2_5_3.

Issue(s) addressed

Link the issues to be closed with this PR, whether in this repository, or in another repository.
None

Testing

How were these changes tested?

Full RTs are run in HAFS model level without any problem
S2S is tested externally by Denise and it is bit-to-bit reproducible
HAFS fully coupled application is also tested through the use of HAFS application and it was Bir-to-bit reproducible
The available data model regression tests are run without any problem: DATM+DOCN, DATM+HYCOM and FV3+DOCN(regional).
NCAR's CESM full test suite also run and all tests were passed - tests only CMEPS and CDEPS

What compilers / HPCs was it tested with?
The entire model is tested on Orion using following HPC-stack.

module load hpc/1.1.0
module load hpc-intel/2018.4
module load hpc-impi/2018.4

S2S is also tested on NCAR's Cheyenne platform.

Are the changes covered by regression tests? (If not, why? Do new tests need to be added?)
No. We are working on following RTs,

FV3 standalone
FV3+HYCOM
FV3+CDEPS DOCN(global, oisst)
FV3+CDEPS DOCN(regional, mom6)
CDEPS DATM(global, era5)+HYCOM
In this case, relatively low resolution regional FV3 configuration (13 km) is used.

Have regression tests and unit tests (utests) been run? On which platforms and with which compilers? (Note that unit tests can only be run on tier-1 platforms)
Full RT run on Orion.

Dependencies

If testing this branch requires non-default branches in other repositories, list them.
Those branches should have matching names (ideally)

Do PRs in upstream repositories need to be merged first?
None

uturuncoglu · 2021-03-12T18:25:29Z

@BinLiu-NOAA Okay. I have just sync with support/HAFS branch. Yes, I have already run full RT without any problem. The new regression tests are almost ready but Orion is overloaded and I could not test the new configurations quickly. This requires little bit more time. Probably, it will be ready on Monday if Orion allow me to work and test them. So, if you want we could wait little bit more until HAFS specific tests are ready and then we could merge this PR. Let me know what do you think? We also coordinate putting new folders to common REGRESSION_TEST folder.

uturuncoglu · 2021-03-12T19:01:34Z

@BinLiu-NOAA I'll solve the log file conflict after the call.

BinLiu-NOAA · 2021-03-12T22:23:03Z

 module load hdf5/1.10.6
 module load netcdf/4.7.4
-module load pio/2.5.1
+module load pio/2.5.2


Update pio/2.5.2 for all platforms; also check fv3_debug

danrosen25

I have code maintainability suggestions for 'CMakeLists.txt'.

danrosen25 · 2021-03-15T16:57:58Z

 if(DATM)
  add_subdirectory(DATM)
-else()
+elseif(NOT CDEPS_DATM)


Cleanup ATM if block. The separate blocks are harder to maintain. The whole section can be labeled ATM
Remove block at 270 'CDEPS Component"
e.g.

if(DATM) add_subdirectory(DATM) elseif(CDEPS_DATM) add_subdirectory(CDEPS) else() add_subdirectory(FV3) endif()

danrosen25 · 2021-03-15T17:03:56Z

  list(APPEND _ufs_defs_private FRONT_DATM=datm)
  add_dependencies(ufs datatm)
-else()
+elseif(NOT CDEPS_DATM)


Move dependencies from below to this section. Either keep the dependency section completely separate from the linking section or combine the entire sections.

346

360

375

I just follow their convention. The existing build was defining them in two operate place. Of course, it will be more efficient with your suggestion but I would rather to keep the convention same. Also, the build system requires major work to make it more flexible and generic.

danrosen25 · 2021-03-15T17:05:24Z

                                        NetCDF::NetCDF_Fortran
                                        ${MKL_LIB})

+if(S2S OR CMEPS OR CDEPS)


Is this only true for PNETCDF used with S2S, CMEPS, or CDEPS or do you mean for these links to be included if PNETCDF is true?

PNETCDF is only required when CMEPS and CDEPS is used. Plus, the io type need to be set to pnetcdf in the nems.configure. These changes will not be used in the near future since it requires new PIO installation with pnetcdf support and the FindPnetCDF.cmake to find the pnetcdf. This is currently in a branch of CMakeModules.

…afs-community/ufs-weather-model into feature/hafs_couplehycom_cdeps

…couplehycom_cdeps

…an support

uturuncoglu · 2021-06-15T15:53:30Z

I think that this is outdated. I am closing this PR for now.

…ces and turning off output; update FAQ documentation (was 1608); update drag suite intent mods (was 1612) (ufs-community#1597) * update cdeps * use fv3atm from PR 1612 * Changed UGWP diagnostic variable declaration intents from out to inout * Docs/faqupdate (#8) Co-authored-by: Denise Worthen <denise.worthen@noaa.gov> Co-authored-by: jkbk2004 <jong.kim@noaa.gov> Co-authored-by: Brian Curtis <brian.curtis@noaa.gov>

Ufuk Turuncoglu added 30 commits January 8, 2021 15:20

enable CDEPS build

abe96d5

disable PIO initialization in CMEPS level, NEMS will do it

34e7097

update compile script to enable data components

23853d4

update CDEPS to enable data components under cmake

483f627

fix PIO issue

6c9bfcd

remove INTERNAL_PIO_INIT. initialization is moved to NEMS

910544f

fix HYCOM cmake to be compatible with feature/NUOPC_merged branch

01198e6

more fix for PIO issue

d71f0f2

fix run seq for datm+docn case to be compatible with updated CMEPS

108d6b8

update the components

996fb14

update runseq for datm+hycom case

754cc11

rename RT configuration file

20de24d

update CDEPS

1b4c3bb

sync components with feature/hafs_couplehycom

4c4ed07

sync model with feature/hafs_couplehycom

caac8e7

remove unused RT configuration file

a9d52fa

fix PIO initialization for S2S

1d63b90

more PIO initialization related fix for S2S configurations

1690f38

fix issue related to data model regression tests

3960577

more fix for RT tests

deb246c

set CDEPS data components as false for other tests

dd2ff5c

update components that has PIO fix

ef98508

clean unused command

f800fab

update HAFS RT tests

e6bcb48

update components

3b73e72

update NEMS to include PIO fix

f6832e1

sync with support/HAFS

2aaa309

fix build for custom DATM configuration tha uses MOM6

d58071d

fix for input.nml in DATM configurations

8e463c8

increase wallclock time from 30 to 45 for coupled configurations

db3d4c8

uturuncoglu marked this pull request as ready for review March 12, 2021 18:25

BinLiu-NOAA changed the title ~~Bring component level PIO initialization~~ Bring in CDEPS and CMEPS/CDEPS component level PIO initialization Mar 12, 2021

BinLiu-NOAA reviewed Mar 12, 2021

View reviewed changes

BinLiu-NOAA requested a review from danrosen25 March 12, 2021 22:42

Update RegressionTests_orion.intel.log

7aa9cd2

danrosen25 requested changes Mar 15, 2021

View reviewed changes

Ufuk Turuncoglu added 11 commits March 16, 2021 11:46

initial attempt to bring HAFS related RTs

b1b64f6

add missing files

544f15c

delete remaining files

d2b9c43

Merge branch 'feature/hafs_couplehycom_cdeps' of https://github.com/h…

31dab3d

…afs-community/ufs-weather-model into feature/hafs_couplehycom_cdeps

minor fix for nemsio

1d673e1

more work for HAFS RTs

0352aed

more work to bring HAFS RTs

040cb1d

add missing file

d17f5a8

remove all data component test

2846a22

update .gitmodules to point feature/pio_fix_comp branches

f38f619

increase wallclock time for DATM+HYCOM RT

48b24db

uturuncoglu changed the title ~~Bring in CDEPS and CMEPS/CDEPS component level PIO initialization~~ Bring in CDEPS, CMEPS/CDEPS component level PIO initialization and add HAFS specific RTs Mar 22, 2021

Ufuk Turuncoglu added 7 commits March 23, 2021 11:29

add environment variables for ESMF timing

9ff89f5

update git submodules

5c70d5e

update components and sync with support/HAFS

845c9ec

Merge remote-tracking branch 'origin/support/HAFS' into feature/hafs_…

41b5242

…couplehycom_cdeps

revert chnages in the CDEPS interface

94628f2

revert chnages related with using full CDEPS library and add data oce…

4aa9759

…an support

update components, point HAFS fork for CDEPS and fix build issue in NEMS

4179b7d

uturuncoglu closed this Jun 15, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Bring in CDEPS, CMEPS/CDEPS component level PIO initialization and add HAFS specific RTs#8

Bring in CDEPS, CMEPS/CDEPS component level PIO initialization and add HAFS specific RTs#8
uturuncoglu wants to merge 58 commits into
support/HAFSfrom
feature/hafs_couplehycom_cdeps

uturuncoglu commented Mar 10, 2021 •

edited

Loading

Uh oh!

uturuncoglu commented Mar 12, 2021

Uh oh!

uturuncoglu commented Mar 12, 2021

Uh oh!

BinLiu-NOAA Mar 12, 2021

Uh oh!

danrosen25 left a comment

Uh oh!

Uh oh!

danrosen25 Mar 15, 2021

Uh oh!

danrosen25 Mar 15, 2021

Uh oh!

uturuncoglu Mar 16, 2021

Uh oh!

danrosen25 Mar 15, 2021

Uh oh!

uturuncoglu Mar 16, 2021

Uh oh!

Uh oh!

Uh oh!

uturuncoglu commented Jun 15, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Conversation

uturuncoglu commented Mar 10, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issue(s) addressed

Testing

Dependencies

Uh oh!

uturuncoglu commented Mar 12, 2021

Uh oh!

uturuncoglu commented Mar 12, 2021

Uh oh!

BinLiu-NOAA Mar 12, 2021

Choose a reason for hiding this comment

Uh oh!

danrosen25 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

danrosen25 Mar 15, 2021

Choose a reason for hiding this comment

Uh oh!

danrosen25 Mar 15, 2021

Choose a reason for hiding this comment

Uh oh!

uturuncoglu Mar 16, 2021

Choose a reason for hiding this comment

Uh oh!

danrosen25 Mar 15, 2021

Choose a reason for hiding this comment

Uh oh!

uturuncoglu Mar 16, 2021

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

uturuncoglu commented Jun 15, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

uturuncoglu commented Mar 10, 2021 •

edited

Loading