Use nccmp to compare netcdf files in regression test on NOAA R&D machines. Update WW3 by DusanJovic-NOAA · Pull Request #1702 · ufs-community/ufs-weather-model

DusanJovic-NOAA · 2023-04-06T15:55:01Z

Description

Update rt_util.sh to use nccmp utility to compare netcdf outputs against the baseline instead of using custom python script. Currently only on NOAA R&D machines (hera, orion, gaea and jet).

Fixes #1657

Top of commit queue on: TBD

Input data additions/changes

No changes are expected to input data.
There will be new input data.
Input data will be updated.

Anticipated changes to regression tests:

No changes are expected to any regression test.
Changes are expected to the following tests:

Subcomponents involved:

Combined with PR's (If Applicable):

Commit Queue Checklist:

Link PR's from all sub-components involved
Confirm reviews completed in sub-component PR's
Add all appropriate labels to this PR.
Run full RT suite on either Hera/Cheyenne with both Intel/GNU compilers
Add list of any failed regression tests to "Anticipated changes to regression tests" section.

Linked PR's and Issues:

Testing Day Checklist:

This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.

Testing Log (for CM's):

RDHPCS
- Intel
  - Hera
  - Orion
  - Jet
  - Gaea
  - Cheyenne
- GNU
  - Hera
  - Cheyenne
WCOSS2
- Dogwood/Cactus
- Acorn
CI
- Completed
opnReqTest
- N/A
- Log attached to comment

DeniseWorthen · 2023-04-06T16:03:13Z

@DusanJovic-NOAA On Cheyenne, nccmp/1.8.2.1 is available. Is there any reason not to include that platform also?

DavidHuber-NOAA · 2023-04-06T16:07:34Z

S4 also has nccmp/1.8.9.0 and I'm happy to test it there.

DusanJovic-NOAA · 2023-04-06T16:49:05Z

I do not have access to cheyenne and S4, so I didn't know whether nccmp is available and which module and how should be loaded. I didn't want to break anything that currently works. Please feel free to make the required changes and test it. I'll then include your changes.

junwang-noaa · 2023-04-07T13:33:31Z

@DusanJovic-NOAA do you see any RT test time change with this update?

jkbk2004 · 2023-04-07T13:59:50Z

@FernandoAndrade-NOAA can you test this pr to confirm nccmp on cheyenne?

DavidHuber-NOAA · 2023-04-07T14:25:34Z

S4 has nccmp installed under hpc-stack, which would require loading the compilers, etc. I have requested it to be installed in a more easily accessible location, but it may take some time. I will include it here before the PR closes if possible, but otherwise I will add it in at a later time.

DusanJovic-NOAA · 2023-04-07T14:48:37Z

@DusanJovic-NOAA do you see any RT test time change with this update?

No significant change in RT test time. For example cpld_control_qr_p8 test which does not produce bit-by-bit identical restart files and must use alternative comparison of netcdf files currently runs in about 355 seconds:

ufs-weather-model/tests/RegressionTests_orion.intel.log

Line 379 in b6146cd

0: The total amount of wall time = 355.689583

while my test runs in about 350 seconds:

  0: The total amount of wall time                        = 350.140382
  0: The maximum resident set size (KB)                   = 3197440

Test 005 cpld_control_qr_p8 PASS

on Orion.

FernandoAndrade-NOAA · 2023-04-07T18:08:46Z

@jkbk2004 adding Cheyenne to the checks seems fine:

0:The total amount of wall time                        = 327.542930
0:The maximum resident set size (KB)                   = 2719748

Test 001 cpld_control_qr_p8 PASS

DusanJovic-NOAA · 2023-04-10T18:14:34Z

@jkbk2004 adding Cheyenne to the checks seems fine:

0:The total amount of wall time                        = 327.542930
0:The maximum resident set size (KB)                   = 2719748

Test 001 cpld_control_qr_p8 PASS

@FernandoAndrade-NOAA Did you run the test on Cheyenne using nccmp? What changes did you make? Can you open PR to my branch or point me to your branch. Or just post the diffs here in the PR. Or just tell what I need to change in order to use nccmp on Cheyenne. Thanks

FernandoAndrade-NOAA · 2023-04-10T18:34:57Z

@jkbk2004 adding Cheyenne to the checks seems fine:
0:The total amount of wall time                        = 327.542930
0:The maximum resident set size (KB)                   = 2719748

Test 001 cpld_control_qr_p8 PASS
@FernandoAndrade-NOAA Did you run the test on Cheyenne using nccmp? What changes did you make? Can you open PR to my branch or point me to your branch. Or just post the diffs here in the PR. Or just tell what I need to change in order to use nccmp on Cheyenne. Thanks

@DusanJovic-NOAA yes on Cheyenne and Jet. For Cheyenne, no major changes I just added Cheyenne to the MACHINE_ID if statements in rt_utils.sh (line 361) and run_test.sh (line 109)

DusanJovic-NOAA · 2023-04-10T18:47:32Z

@jkbk2004 adding Cheyenne to the checks seems fine:
0:The total amount of wall time                        = 327.542930
0:The maximum resident set size (KB)                   = 2719748

Test 001 cpld_control_qr_p8 PASS
@FernandoAndrade-NOAA Did you run the test on Cheyenne using nccmp? What changes did you make? Can you open PR to my branch or point me to your branch. Or just post the diffs here in the PR. Or just tell what I need to change in order to use nccmp on Cheyenne. Thanks
@DusanJovic-NOAA yes on Cheyenne and Jet. For Cheyenne, no major changes I just added Cheyenne to the MACHINE_ID if statements in rt_utils.sh (line 361) and run_test.sh (line 109)

Thanks

jkbk2004 · 2023-04-21T13:52:31Z

@DusanJovic-NOAA can we combine in #1717 to this pr? We just need to point to ww3 branch https://github.com/jessicameixner-noaa/WW3/tree/feature/syncWW30419. Once combined, I like to ask to modify the PR title to reflect #1717.

DusanJovic-NOAA · 2023-04-21T14:55:23Z

@DusanJovic-NOAA can we combine in #1717 to this pr? We just need to point to ww3 branch https://github.com/jessicameixner-noaa/WW3/tree/feature/syncWW30419. Once combined, I like to ask to modify the PR title to reflect #1717.

Done.

FernandoAndrade-NOAA · 2023-04-21T16:48:05Z

@BrianCurtis-NOAA @JessicaMeixner-NOAA @MatthewMasarik-NOAA We're going to begin testing for this PR

jkbk2004 · 2023-04-24T15:15:57Z

Gaea C4 is not available these days

The paths in the error message are not GAEA paths. They look like WCOSS2 or Acorn to me:

baseline dir = /lfs/h2/emc/nems/noscrub/emc.nems/RT/NEMSfv3gfs/develop-20230418/INTEL/atmaero_control_p8
working dir = /lfs/h2/emc/ptmp/brian.curtis/FV3_RT/rt_66056/atmaero_control_p8

Yeah, @BrianCurtis-NOAA test error is from wcoss2.

BrianCurtis-NOAA · 2023-04-24T15:21:10Z

Yes WCOSS2:

brian.curtis@clogin02:/lfs/h2/emc/nems/noscrub/brian.curtis/git/DusanJovic-NOAA/ufs-weather-model/tests> 
module load python/3.8.6
brian.curtis@clogin02:/lfs/h2/emc/nems/noscrub/brian.curtis/git/DusanJovic-NOAA/ufs-weather-model/tests> ./compare_ncfile.py /lfs/h2/emc/ptmp/brian.curtis/FV3_RT/rt_66056/atmaero_control_p8/RESTART/20210323.060000.phy_data.tile5.nc /lfs/h2/emc/nems/noscrub/emc.nems/RT/NEMSfv3gfs/develop-20230418/INTEL/atmaero_control_p8/RESTART/20210323.060000.phy_data.tile5.nc
Traceback (most recent call last):
  File "./compare_ncfile.py", line 6, in <module>
    with Dataset(sys.argv[1]) as nc1, Dataset(sys.argv[2]) as nc2:
  File "src/netCDF4/_netCDF4.pyx", line 2330, in netCDF4._netCDF4.Dataset.__init__
  File "src/netCDF4/_netCDF4.pyx", line 1948, in netCDF4._netCDF4._ensure_nc_success
OSError: [Errno -51] NetCDF: Unknown file format: b'/lfs/h2/emc/nems/noscrub/emc.nems/RT/NEMSfv3gfs/develop-20230418/INTEL/atmaero_control_p8/RESTART/20210323.060000.phy_data.tile5.nc'

This is the second straight failure for this test.

DusanJovic-NOAA · 2023-04-24T15:26:10Z

That file is empty:

clogin01:/u/dusan.jovic> ls -l /lfs/h2/emc/nems/noscrub/emc.nems/RT/NEMSfv3gfs/develop-20230418/INTEL/atmaero_control_p8/RESTART/
total 1427328
-rw-r--r-- 1 emc.nems nems       300 Apr 19 19:23 20210323.060000.coupler.res
-rw-r--r-- 1 emc.nems nems     18784 Apr 19 19:23 20210323.060000.fv_core.res.nc
-rw-r--r-- 1 emc.nems nems  37610116 Apr 19 19:23 20210323.060000.fv_core.res.tile1.nc
-rw-r--r-- 1 emc.nems nems  37610116 Apr 19 19:23 20210323.060000.fv_core.res.tile2.nc
-rw-r--r-- 1 emc.nems nems  37610116 Apr 19 19:23 20210323.060000.fv_core.res.tile3.nc
-rw-r--r-- 1 emc.nems nems  37610116 Apr 19 19:23 20210323.060000.fv_core.res.tile4.nc
-rw-r--r-- 1 emc.nems nems  37610116 Apr 19 19:23 20210323.060000.fv_core.res.tile5.nc
-rw-r--r-- 1 emc.nems nems  37610116 Apr 19 19:23 20210323.060000.fv_core.res.tile6.nc
-rw-r--r-- 1 emc.nems nems     92200 Apr 19 19:23 20210323.060000.fv_srf_wnd.res.tile1.nc
-rw-r--r-- 1 emc.nems nems     92200 Apr 19 19:23 20210323.060000.fv_srf_wnd.res.tile2.nc
-rw-r--r-- 1 emc.nems nems     92200 Apr 19 19:23 20210323.060000.fv_srf_wnd.res.tile3.nc
-rw-r--r-- 1 emc.nems nems     92200 Apr 19 19:23 20210323.060000.fv_srf_wnd.res.tile4.nc
-rw-r--r-- 1 emc.nems nems     92200 Apr 19 19:23 20210323.060000.fv_srf_wnd.res.tile5.nc
-rw-r--r-- 1 emc.nems nems     92200 Apr 19 19:23 20210323.060000.fv_srf_wnd.res.tile6.nc
-rw-r--r-- 1 emc.nems nems 163883040 Apr 19 19:23 20210323.060000.fv_tracer.res.tile1.nc
-rw-r--r-- 1 emc.nems nems 163883040 Apr 19 19:23 20210323.060000.fv_tracer.res.tile2.nc
-rw-r--r-- 1 emc.nems nems 163883040 Apr 19 19:23 20210323.060000.fv_tracer.res.tile3.nc
-rw-r--r-- 1 emc.nems nems 163883040 Apr 19 19:23 20210323.060000.fv_tracer.res.tile4.nc
-rw-r--r-- 1 emc.nems nems 163883040 Apr 19 19:23 20210323.060000.fv_tracer.res.tile5.nc
-rw-r--r-- 1 emc.nems nems 163883040 Apr 19 19:23 20210323.060000.fv_tracer.res.tile6.nc
-rw-r--r-- 1 emc.nems nems  38950016 Apr 19 19:23 20210323.060000.phy_data.tile1.nc
-rw-r--r-- 1 emc.nems nems  38950016 Apr 19 19:23 20210323.060000.phy_data.tile2.nc
-rw-r--r-- 1 emc.nems nems  38950016 Apr 19 19:23 20210323.060000.phy_data.tile3.nc
-rw-r--r-- 1 emc.nems nems  38950016 Apr 19 19:23 20210323.060000.phy_data.tile4.nc
-rw-r--r-- 1 emc.nems nems         0 Apr 19 19:23 20210323.060000.phy_data.tile5.nc
-rw-r--r-- 1 emc.nems nems  38950016 Apr 19 16:10 20210323.060000.phy_data.tile6.nc
-rw-r--r-- 1 emc.nems nems   9538756 Apr 19 16:10 20210323.060000.sfc_data.tile1.nc
-rw-r--r-- 1 emc.nems nems   9538756 Apr 19 16:10 20210323.060000.sfc_data.tile2.nc
-rw-r--r-- 1 emc.nems nems   9538756 Apr 19 16:10 20210323.060000.sfc_data.tile3.nc
-rw-r--r-- 1 emc.nems nems   9538756 Apr 19 16:10 20210323.060000.sfc_data.tile4.nc
-rw-r--r-- 1 emc.nems nems   9538756 Apr 19 16:10 20210323.060000.sfc_data.tile5.nc
-rw-r--r-- 1 emc.nems nems   9538756 Apr 19 16:10 20210323.060000.sfc_data.tile6.nc

BrianCurtis-NOAA · 2023-04-24T15:27:41Z

Why would one tile not write?

BrianCurtis-NOAA · 2023-04-24T15:28:46Z

OK, well. I will redo that baseline again from develop branch. Then run the test again. Thanks for the catch @DusanJovic-NOAA

SamuelTrahanNOAA · 2023-04-24T15:34:58Z

Why would one tile not write?

Three thoughts:

Was that the baseline you generated, or was it copied from Dogwood? If it is copied, make sure you use rsync, and run rsync twice. The second run will confirm that there is nothing left to transfer. Don't ever use scp; it is not fault tolerant.
Do you compare against the baseline that you generate for a PR? That should have caught a zero-sized baseline file.
Do you have the logs from the baseline generation? It may give us clues as to why the copy failed. (Unless the baseline was copied from another machine.)

BrianCurtis-NOAA · 2023-04-24T15:37:25Z

Why would one tile not write?

Three thoughts:

Was that the baseline you generated, or was it copied from Dogwood? If it is copied, make sure you use rsync, and run rsync twice. The second run will confirm that there is nothing left to transfer. Don't ever use scp; it is not fault tolerant.

Do you compare against the baseline that you generate for a PR? That should have caught a zero-sized baseline file.

Do you have the logs from the baseline generation? It may give us clues as to why the copy failed. (Unless the baseline was copied from another machine.)

With all the switches, probably 1, since 2 is Yes and 3 is Yes. I'm not sure what @RatkoVasic-NOAA does, but it may be worth at least double checking his CRON/script.

SamuelTrahanNOAA · 2023-04-24T15:45:01Z

With all the switches, probably 1, since 2 is Yes and 3 is Yes. I'm not sure what @RatkoVasic-NOAA does, but it may be worth at least double checking his CRON/script.

You should also avoid "cp" or "mv" within the same machine. A large filesystem operation can be unreliable. Rsync is fault tolerant, and it will confirm the transfer was complete on a second run.

The only exception is a "mv" within the same fileset of the same filesystem in the same cluster. In that special case, "mv" is a unit operation. That makes it especially safe.

RatkoVasic-NOAA · 2023-04-24T16:53:38Z

If you are talking about syncing data between two WCOSS2 machines, I'm using rsync in cron. There's a log file for each day in /u/emc.nems/ratko.vasic/202*

RatkoVasic-NOAA · 2023-04-24T16:57:29Z

I see that file with zero size is on both machines.

BrianCurtis-NOAA · 2023-04-24T17:02:39Z

@RatkoVasic-NOAA Thanks for checking on Dogwood! The new baseline was OK and passed testing, so hopefully it was a random issue and not a long term one.

jkbk2004 · 2023-04-24T17:44:00Z

@JessicaMeixner-NOAA all tests are done. Please, go ahead to merge ww3 PR.

JessicaMeixner-NOAA · 2023-04-24T18:01:20Z

WW3 has been merged. New hash is e026bcc

DusanJovic-NOAA added 4 commits April 4, 2023 10:41

Use nccmp to compare netcdf files

2aabc7e

Merge remote-tracking branch 'origin/develop' into rt_nccmp

0489f7c

Use nccmp only on NOAA R&D systems

37eb6ec

Do not skip checking fv_core.res.nc when GNU compiler is used

db63d9d

DusanJovic-NOAA added the No Baseline Change No Baseline Change label Apr 6, 2023

DusanJovic-NOAA requested a review from BrianCurtis-NOAA April 6, 2023 15:55

DusanJovic-NOAA added 2 commits April 7, 2023 11:10

Update ufs_common.lua

440333f

Update ufs_hera modulefiles

7daab87

Use nccmp on Cheyenne

648f928

DusanJovic-NOAA added 2 commits April 10, 2023 18:48

Cleanup compile.sh. Remove unnecessary FIXME for gnu compiler

1f3b85b

Merge remote-tracking branch 'origin/develop' into rt_nccmp

172cfa3

jkbk2004 mentioned this pull request Apr 21, 2023

Use optional chunksizes argument in register_restart_field calls (combining PRs #1853 & 1834) #1467

Merged

16 tasks

Update WW3

00bcdd7

DusanJovic-NOAA changed the title ~~Use nccmp to compare netcdf files in regression test on NOAA R&D machines~~ Use nccmp to compare netcdf files in regression test on NOAA R&D machines. Update WW3 Apr 21, 2023

FernandoAndrade-NOAA added Ready for Commit Queue The PR is ready for the Commit Queue. All checkboxes in PR template have been checked. jenkins-ci Jenkins CI: ORT build/test on docker container labels Apr 21, 2023

WCOSS2 Intel RT Log

75a5bda

jkbk2004 requested review from DeniseWorthen, MatthewMasarik-NOAA, jkbk2004 and zach1221 April 24, 2023 17:25

DeniseWorthen previously approved these changes Apr 24, 2023

View reviewed changes

zach1221 previously approved these changes Apr 24, 2023

View reviewed changes

Update WW3

a887cce

DusanJovic-NOAA dismissed stale reviews from zach1221 and DeniseWorthen via a887cce April 24, 2023 18:05

jkbk2004 approved these changes Apr 24, 2023

View reviewed changes

zach1221 approved these changes Apr 24, 2023

View reviewed changes

FernandoAndrade-NOAA merged commit 46ef6d5 into ufs-community:develop Apr 24, 2023

Jili-Dong mentioned this pull request Apr 24, 2023

Use nccmp to compare netcdf files in regression test on NOAA R&D mach… Jili-Dong/ufs-weather-model#6

Merged

37 tasks

jkbk2004 mentioned this pull request Apr 24, 2023

Update WW3 #1717

Closed

37 tasks

grantfirl mentioned this pull request Jun 13, 2023

UFS-dev PR#26 NCAR/ufs-weather-model#98

Merged

DusanJovic-NOAA deleted the rt_nccmp branch August 31, 2023 20:46

Conversation

DusanJovic-NOAA commented Apr 6, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Top of commit queue on: TBD

Input data additions/changes

Anticipated changes to regression tests:

Subcomponents involved:

Combined with PR's (If Applicable):

Commit Queue Checklist:

Linked PR's and Issues:

Testing Day Checklist:

Testing Log (for CM's):

Uh oh!

DeniseWorthen commented Apr 6, 2023

Uh oh!

DavidHuber-NOAA commented Apr 6, 2023

Uh oh!

DusanJovic-NOAA commented Apr 6, 2023

Uh oh!

junwang-noaa commented Apr 7, 2023

Uh oh!

jkbk2004 commented Apr 7, 2023

Uh oh!

DavidHuber-NOAA commented Apr 7, 2023

Uh oh!

DusanJovic-NOAA commented Apr 7, 2023

Uh oh!

FernandoAndrade-NOAA commented Apr 7, 2023

Uh oh!

DusanJovic-NOAA commented Apr 10, 2023

Uh oh!

FernandoAndrade-NOAA commented Apr 10, 2023

Uh oh!

DusanJovic-NOAA commented Apr 10, 2023

Uh oh!

jkbk2004 commented Apr 21, 2023

Uh oh!

DusanJovic-NOAA commented Apr 21, 2023

Uh oh!

FernandoAndrade-NOAA commented Apr 21, 2023

Uh oh!

jkbk2004 commented Apr 24, 2023

Uh oh!

BrianCurtis-NOAA commented Apr 24, 2023

Uh oh!

DusanJovic-NOAA commented Apr 24, 2023

Uh oh!

BrianCurtis-NOAA commented Apr 24, 2023

Uh oh!

BrianCurtis-NOAA commented Apr 24, 2023

Uh oh!

SamuelTrahanNOAA commented Apr 24, 2023

Uh oh!

BrianCurtis-NOAA commented Apr 24, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

SamuelTrahanNOAA commented Apr 24, 2023

Uh oh!

RatkoVasic-NOAA commented Apr 24, 2023

Uh oh!

RatkoVasic-NOAA commented Apr 24, 2023

Uh oh!

BrianCurtis-NOAA commented Apr 24, 2023

Uh oh!

jkbk2004 commented Apr 24, 2023

Uh oh!

JessicaMeixner-NOAA commented Apr 24, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

13 participants

DusanJovic-NOAA commented Apr 6, 2023 •

edited

Loading

BrianCurtis-NOAA commented Apr 24, 2023 •

edited

Loading