Update ufs-weather for latest CICE emc/develop; Update to esmf-bs47; add export of openwater_frac_in_atm from fv3ATM when coupled by DeniseWorthen · Pull Request #475 · ufs-community/ufs-weather-model

DeniseWorthen · 2021-03-14T14:47:54Z

Description

This PR will:

update CICE in ufs-weather for the latest emc/develop at NOAA-EMC/CICE.
update fv3ATM to use esmf-bs47, add the export of open-water fraction from fv3ATM and update module files across all platforms to use bs47 (PR Update to esmf-bs47 across all platforms; add export of openwater_frac_in_atm from FV3 when coupled #493)

Issue(s) addressed

CICE Issue #25
UFS Issue Update ESMF to bs47 across all platforms #494
partial fix to issue UFS the coupled model requires FV3 and CMEPS updates for frac grid normalization #366. FV3atm will export the required field but no use will be made of it until changes are made in the CMEPS mediator.

Testing

This PR will change all ufs-coupled and ufs-datm baselines due to a non-b4b bugfix in Icepack. A test baseline was created on hera.intel and all cpld and datm regression tests passed against that test baseline.
This PR also changes baselines due to the change to esmf-bs47 because some forecast files will contain different metadata or different ordering of fields. See fv3ATM Issue #265 for more details on the changes expected.

Dependencies

NEMS PR#94
CICE PR #24
CMEPS PR #39
FV3-ATM PR #266

use updcmeps branch of CMEPS-interface/CMEPS add file to CMakeLists shorten name of nems.configure file for coupled model clean up white space

add rahul's fix for optionally loading fv3_debug if exists and debug=y

add file to cice_files.cmake update CICE with use statement added to CICEInitMod

skip-ci

add regression test log showing that debug test passes baseline cpld_control fails due to refactor

* Add $INPUTDATA_ROOT * Update utest * rt.sh: Remove LOCKDIR before trying to stop ecflow in cleanup()

DeniseWorthen · 2021-04-02T14:26:11Z

@junwang-noaa all the submodules have approvals at this point except CICE--can you please review/approve?

DeniseWorthen · 2021-04-02T15:53:17Z

I put the jet baseline in the wrong location for the auto-rt to find it (/lfs4/HFIP/hfv3gfs/RT/NEMSfv3gfs vs /lfs4/HFIP/hfv3gfs/emc.nemspara/RT/NEMSfv3gfs) so we'll be restarting the jet auto-rt.

BrianCurtis-NOAA · 2021-04-02T18:09:17Z

Machine: jet
Compiler: intel
Repo location: /lfs4/HFIP/h-nems/emc.nemspara/autort/pr/592556664/20210402160009/ufs-weather-model
Please manually delete: /lfs4/HFIP/hfv3gfs/emc.nemspara/RT_RUNDIRS/emc.nemspara/FV3_RT/rt_18641
Test fv3_stretched_nest 019 failed in run_test failed
Test fv3_HAFS_v0_hwrf_thompson 046 failed in check_result failed
Test fv3_HAFS_v0_hwrf_thompson 046 failed in run_test failed
Please make changes and add the following label back:
jet-intel-RT

DeniseWorthen · 2021-04-02T18:31:30Z

The fv3_stretched_nest failed due to time wall clock but it actually seemed to have hung. It is a 48hr forecast and it looks like it only got a few timesteps.

The HAFS test failed on the 4x daily tile2 because the baseline file has no time level information written:

time = UNLIMITED ; // (0 currently)

The other 4x daily files for the other tiles have 4 timesteps correctly written and they compare identically with the verify run.

I will add that I'm having some trouble getting response from jet when I ls or cd. So perhaps there is a system issue going on here too.

DeniseWorthen · 2021-04-03T10:21:25Z

The baselines on jet are not completing (compiling) because the disk is full:

CMakeFiles/ccppphys.dir/physics/GFS_phys_time_vary.fv3.F90.o: catastrophic error: Disk is full, no space to write object file

This is in my own attempt to run the last two baselines as well as the auto-rt I started late yesterday.

BrianCurtis-NOAA · 2021-04-05T02:40:22Z

Machine: jet
Compiler: intel
Repo location: /lfs4/HFIP/h-nems/emc.nemspara/autort/pr/592556664/20210402230015/ufs-weather-model
Please manually delete: /lfs4/HFIP/hfv3gfs/emc.nemspara/RT_RUNDIRS/emc.nemspara/FV3_RT/rt_150647
Test compile 014 failed failed
Test compile 013 failed failed
Test fv3_cpt 032 failed in run_test failed
Test fv3_wrtGauss_nemsio_c192 010 failed in run_test failed
Test fv3_gfs_v16_RRTMGP_2thrd 041 failed in run_test failed
Test fv3_esg_HAFS_v0_hwrf_thompson 047 failed in run_test failed
Please make changes and add the following label back:
jet-intel-RT

* update CICE, CMEPS and NEMS submodules

DeniseWorthen · 2021-04-05T14:30:23Z

Ready for review & merge.

climbfuji · 2021-04-05T14:35:27Z

 RUN     | fv3_regional_control                                                                                                    |                                         | fv3 |
 RUN     | fv3_regional_restart                                                                                                    |                                         | fv3 | fv3_regional_control
-RUN     | fv3_regional_quilt                                                                                                      |                                         | fv3 |
+#RUN     | fv3_regional_quilt                                                                                                      |                                         | fv3 |


Do we have a timeline for fixing this?

I think Jun has tracked down the issues here. She's opened an associated issue in EMC_post.

climbfuji · 2021-04-05T14:35:54Z

 export_fv3

 export TASKS=192
+export WLCLK=60


These longer runtimes need to be investigated.

A full 60 minutes is not required but this is what I tested out of frustration. The runs ended up taking about 20-25 minutes in my case.

MinsukJi-NOAA · 2021-04-05T15:03:40Z

@DeniseWorthen could you include these changes? It fixes the esmf bs47 docker container issue.

diff --git a/tests/ci/Dockerfile b/tests/ci/Dockerfile
index c685eaf..8d01299 100644
--- a/tests/ci/Dockerfile
+++ b/tests/ci/Dockerfile
@@ -1,4 +1,4 @@
-From noaaemc/ubuntu-hpc:v1.1
+From noaaemc/ubuntu-hpc:v1.2
 
 ENV HOME=/home/builder
 COPY --chown=builder:builder . $HOME/ufs-weather-model

DeniseWorthen · 2021-04-05T15:52:14Z

@MinsukJi-NOAA Should I now try run-ci?

MinsukJi-NOAA · 2021-04-05T15:54:11Z

Yes, that would be great.

DeniseWorthen · 2021-04-05T16:52:30Z

@MinsukJi-NOAA Two CI tests failed. Should I hold off merging until you can take a look?

MinsukJi-NOAA · 2021-04-05T17:02:45Z

I will take a look. I suggest you go ahead with merging for this PR.

@SMoorthi-emc

…(combined) (ufs-community#475) * add lsm_cold_start variable in GFS_typedefs.F90/meta and update ccpp/physics for testing * includes @SMoorthi-emc 's bugfix for gwdps.f.

…e-offset (#475) * change PDY to yyyymmdd for time-offset * change prod in nco we2e script * modify ex-script rather than machine file * fix typo * Set PDYext cycext Co-authored-by: chan-hoo <chan-hoo.jeon@clogin01.cactus.wcoss2.ncep.noaa.gov>

DeniseWorthen and others added 30 commits October 28, 2020 12:47

update cmeps

ca288f4

use updcmeps branch of CMEPS-interface/CMEPS add file to CMakeLists shorten name of nems.configure file for coupled model clean up white space

fix for debug mode compile

a21c7e9

add rahul's fix for optionally loading fv3_debug if exists and debug=y

update cmeps

0b655ed

Merge branch 'develop' into feature/updcmeps

1699be7

point to updcice branch

399d21d

updates for ice_dyn_vp

06bad08

add file to cice_files.cmake update CICE with use statement added to CICEInitMod

update to last CMEPS master prior to perf updates

5d51334

update CMEPS submodule

21543a2

Merge branch 'develop' into feature/updcmeps

2017af3

skip-ci

Merge branch 'develop' into feature/updcice

b558028

Merge branch 'feature/updcmeps' into feature/updcice

394c7ad

Merge branch 'develop' into feature/updcice

5d19c0f

update cice branch of ufs-weather

2d12b93

add regression test log showing that debug test passes baseline cpld_control fails due to refactor

update gitmodules and CICE

0e2cc16

set n_aero to zero

e4af1ec

Merge branch 'develop' into feature/updcice

a356d4c

Merge branch 'develop' into feature/updcice

0564cdf

Merge branch 'develop' into feature/updcice

381142b

Merge branch 'develop' into feature/updcice

42fe791

Merge branch 'develop' into feature/updcice

c73dcd9

Merge branch 'develop' into feature/updcice

d320bd5

Merge branch 'develop' into feature/updcice

81b7165

Merge branch 'develop' into feature/updcice skip-ci

e93cfae

add cheyenne.intel for ufs-cpld and ufs-datm skip-ci

ab1ca9f

rt.sh separate input (#2)

541c714

* Add $INPUTDATA_ROOT * Update utest * rt.sh: Remove LOCKDIR before trying to stop ecflow in cleanup()

add hera gnu logs; update baseline date; skip-ci

8950d96

add orion.intel logs; skip-ci

b3d2cf2

add hera.intel log; skip-ci

79205a6

add wcoss-cray log; skip-ci

d8d1d1d

add dell-p3 log; skip-ci

6c760ac

MinsukJi-NOAA mentioned this pull request Apr 2, 2021

How to maintain hpc-stack up to date with ufs-weather-model NOAA-EMC/hpc-stack#205

Closed

BrianCurtis-NOAA added jet-intel-RT and removed jet-intel-RT labels Apr 2, 2021

DeniseWorthen added the jet-intel-RT label Apr 2, 2021

BrianCurtis-NOAA removed the jet-intel-RT label Apr 2, 2021

add jet.intel log

50c1999

DeniseWorthen added 2 commits April 5, 2021 09:02

update submodules

343da16

* update CICE, CMEPS and NEMS submodules

update FV3 submodule

446da5b

DusanJovic-NOAA approved these changes Apr 5, 2021

View reviewed changes

climbfuji approved these changes Apr 5, 2021

View reviewed changes

BrianCurtis-NOAA approved these changes Apr 5, 2021

View reviewed changes

update Dockerfile

5bdae36

run-ci

d067cc4

DeniseWorthen merged commit b8ed338 into ufs-community:develop Apr 5, 2021

This was referenced Apr 5, 2021

CMEPS needs to use imported ocean fraction from FV3 to properly normalize fluxes NOAA-EMC/CMEPS#29

Closed

compare_ncfile error on cray #500

Closed

Conversation

DeniseWorthen commented Mar 14, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Issue(s) addressed

Testing

Dependencies

Uh oh!

DeniseWorthen commented Apr 2, 2021

Uh oh!

DeniseWorthen commented Apr 2, 2021

Uh oh!

BrianCurtis-NOAA commented Apr 2, 2021

Uh oh!

DeniseWorthen commented Apr 2, 2021 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DeniseWorthen commented Apr 3, 2021

Uh oh!

BrianCurtis-NOAA commented Apr 5, 2021

Uh oh!

DeniseWorthen commented Apr 5, 2021

Uh oh!

climbfuji Apr 5, 2021

Choose a reason for hiding this comment

Uh oh!

DeniseWorthen Apr 5, 2021

Choose a reason for hiding this comment

Uh oh!

climbfuji Apr 5, 2021

Choose a reason for hiding this comment

Uh oh!

DeniseWorthen Apr 5, 2021

Choose a reason for hiding this comment

Uh oh!

MinsukJi-NOAA commented Apr 5, 2021

Uh oh!

DeniseWorthen commented Apr 5, 2021

Uh oh!

MinsukJi-NOAA commented Apr 5, 2021

Uh oh!

DeniseWorthen commented Apr 5, 2021

Uh oh!

MinsukJi-NOAA commented Apr 5, 2021

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

DeniseWorthen commented Mar 14, 2021 •

edited

Loading

DeniseWorthen commented Apr 2, 2021 •

edited

Loading