Add zstandard compression to fv3atm; change references from "nems" to "ufs", allow file clobbering for GOCART history files (was #1948) by DusanJovic-NOAA · Pull Request #1932 · ufs-community/ufs-weather-model

DusanJovic-NOAA · 2023-10-04T13:19:53Z

PR Author Checklist:

I have linked PR's from all sub-components involved in section below.
I am confirming reviews are completed in ALL sub-component PR's.
I have run the full RT suite on either Hera/Cheyenne AND have attached the log to this PR below this line:
- LOG:
I have added the list of all failed regression tests to "Anticipated changes" section.
I have filled out all sections of the template.

Description

Add option to enable zstandard compression in fv3atm.

New model_configure configuration parameter has been added (zstandard_level). Valid values are 0-19 (default is 0). Value 0 turns off compression, values 1-19 specify the level of compression (1=fastest compression, 19=best compression ratio),

This PR is combined with #1948 (@DeniseWorthen)

Linked Issues and Pull Requests

Associated UFSWM Issue to close

#1927

Subcomponent Pull Requests

NOAA-EMC/ufsatm#706

depends on change nems to ufs, update coupling modes NOAA-EMC/CMEPS#101

Blocking Dependencies

Subcomponents involved:

Anticipated Changes

Input data

No changes are expected to input data.
Changes are expected to input data:
- New input data.
- Updated input data.

Regression Tests:

No changes are expected to any regression test.
Changes are expected to the following tests:

Tests effected by changes in this PR:

Libraries

Not Needed
Needed
- Create separate issue in JCSDA/spack-stack asking for update to library. Include library name, library version.
- Add issue link from JCSDA/spack-stack following this item

Code Managers Log

This PR is up-to-date with the top of all sub-component repositories except for those sub-components which are the subject of this PR.
Move new/updated input data on RDHPCS Hera and propagate input data changes to all supported systems.
- N/A

Testing Log:

RDHPCS
- Hera
- Orion
- Hercules
- Jet
- Gaea
- Cheyenne
WCOSS2
- Dogwood/Cactus
- Acorn
CI
- Completed
opnReqTest
- N/A
- Log attached to comment

DusanJovic-NOAA · 2023-10-05T13:27:40Z

On Hera the nccmp (version 1.9.0.1 from spack-stack) fails with 'Illegal instruction' error:

baseline dir = /scratch2/NAGAPE/epic/UFS-WM_RT/NEMSfv3gfs/develop-20231002/control_p8_intel
working dir  = /scratch1/NCEPDEV/stmp2/Dusan.Jovic/FV3_RT/rt_308812/control_p8_intel
Checking test 001 control_p8_intel results ....
 Comparing sfcf000.nc ............ALT CHECK..rt_utils.sh: line 324: 30915 Illegal instruction     nccmp -d -S -q -f -g -B --Attribute=checksum --warn=format ${RTPWD}/${CNTL_DIR}_${RT_COMPILER}/${i} ${RUNDIR}/${i} > ${i}_nccmp.log 2>&1
....ERROR

This happens on ecflow node (hecflow01). When I run this test without ecflow (on login node), nccmp runs fine.

DusanJovic-NOAA · 2023-10-05T16:29:29Z

Regression test passed on Hera (using rocoto due to the above issue with nccmp on ecflow nodes) RegressionTests_hera.log

DusanJovic-NOAA · 2023-10-05T16:30:32Z

Regression test passed on Hercules (using ecflow)
RegressionTests_hercules.log

* all except lnd sbs run

This reverts commit 5f38b30.

…fs-weather-model into feature/nems2ufs

* small print changes added to instlnd branch instead

DusanJovic-NOAA · 2023-10-31T14:59:14Z

I see this error in one of the PET error files:

20231031 131945.409 ERROR            PET141 atmos_model.F90:3154 Not found  - Failure to populate exported field: inst_pres_interface
20231031 131945.409 ERROR            PET141 module_fcst_grid_comp.F90:500 Not found  - Passing error in return code
20231031 131945.409 ERROR            PET141 module_fcst_grid_comp.F90:1245 Not found  - Passing error in return code
20231031 131945.409 ERROR            PET141 fv3_cap.F90:1017 Not found  - Passing error in return code
20231031 131945.409 ERROR            PET141 ATM:src/addon/NUOPC/src/NUOPC_ModelBase.F90:978 Not found  - Passing error in return code
20231031 131945.409 ERROR            PET141 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:2838 Not found  - Phase 'IPDvXp04' Initialize for modelComp 2: ATM did not return ESMF_SUCCESS
20231031 131945.409 ERROR            PET141 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:1975 Not found  - Passing error in return code
20231031 131945.409 ERROR            PET141 UFS Driver Grid Comp:src/addon/NUOPC/src/NUOPC_Driver.F90:481 Not found  - Passing error in return code
20231031 131945.409 ERROR            PET141 UFS.F90:386 Not found  - Aborting UFS
20231031 131945.409 INFO             PET141 Finalizing ESMF

DusanJovic-NOAA · 2023-10-31T15:18:28Z

Those three failed tests are still using NEMS_CONFIGURE:

$ grep -r NEMS_CONFIGURE .
./tests/cpld_mpi_gfsv17:export NEMS_CONFIGURE=nems.configure.cpld_noaero_outwav.IN
./tests/cpld_restart_gfsv17:export NEMS_CONFIGURE=nems.configure.cpld_noaero_outwav.IN
./tests/cpld_debug_gfsv17:export NEMS_CONFIGURE=nems.configure.cpld_noaero_outwav.IN

DeniseWorthen · 2023-10-31T15:31:49Z

@DusanJovic-NOAA Thanks. That's because these are added tests, so they didn't have the changes I had made in my PR. Let me fix them.

DeniseWorthen · 2023-10-31T15:43:31Z

@DusanJovic-NOAA Actually it might be easier for you to fix, now that these have been combined. The change in each test should be

-export NEMS_CONFIGURE=nems.configure.cpld_noaero_outwav.IN
+export UFS_CONFIGURE=ufs.configure.cpld_noaero_outwav.IN

DusanJovic-NOAA · 2023-10-31T16:08:47Z

@DeniseWorthen Done. Please double check. 630f983

BrianCurtis-NOAA · 2023-10-31T16:10:47Z

@DeniseWorthen Done. Please double check. 630f983

I made these changes a bit ago and Acorn has finished successfully, can confirm they are the correct changes.

DeniseWorthen · 2023-10-31T16:12:01Z

@DusanJovic-NOAA Looks good, thanks.

…r-model into add_zstandard

zach1221 · 2023-10-31T19:52:35Z

Testing is complete. Let's start the merging process of the sub-component PRs.

DeniseWorthen · 2023-10-31T19:59:16Z

@DusanJovic-NOAA CMEPS has been merged e5d08d4

zach1221 · 2023-10-31T20:25:21Z

@DusanJovic-NOAA fv3atm sub-pr merged. Please revert gitmodule url and update hash. fv3atm hash: 1250b41

junwang-noaa · 2023-10-31T20:33:39Z

@DeniseWorthen would you please create an issue in global-workflow about the nems.configure and file clobber updates when this version of model is updated in GW? Thanks

bbakernoaa · 2023-10-31T21:00:53Z

I had included the clobber update in a PR for the global-workflow here NOAA-EMC/global-workflow#1989

…

On Tue, Oct 31, 2023 at 4:33 PM Jun Wang ***@***.***> wrote: @DeniseWorthen <https://github.com/DeniseWorthen> would you please create an issue in global-workflow about the nems.configure and file clobber updates when this version of model is updated in GW? Thanks — Reply to this email directly, view it on GitHub <#1932 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AFIUVN7K473RJBTPIYJRVBLYCFODBAVCNFSM6AAAAAA5ST4PJ6VHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTOOBXHE4TAMRRHA> . You are receiving this because you are subscribed to this thread.Message ID: ***@***.***>

DeniseWorthen · 2023-10-31T22:25:08Z

The nems->ufs GW issue is NOAA-EMC/global-workflow#1942

DusanJovic-NOAA added 8 commits September 15, 2023 12:11

Update .gitmodules

3f5eb0c

Update FV3

5e160d3

Update FV3

5e4760f

Update FV3

f69df24

Update FV3

6436dca

Merge remote-tracking branch 'origin/develop' into add_zstandard

9335e72

Update FV3

33089ee

Merge remote-tracking branch 'origin/develop' into add_zstandard

47e798d

DusanJovic-NOAA requested a review from junwang-noaa October 4, 2023 13:20

junwang-noaa previously approved these changes Oct 4, 2023

View reviewed changes

Enable zstd compression in several tests

921de47

DusanJovic-NOAA dismissed junwang-noaa’s stale review via 921de47 October 5, 2023 12:55

DusanJovic-NOAA mentioned this pull request Oct 5, 2023

nccmp crashes on Hera ecflow node (hecflow01) JCSDA/spack-stack#812

Closed

DusanJovic-NOAA and others added 3 commits October 5, 2023 16:44

Merge remote-tracking branch 'origin/develop' into add_zstandard

5b8fa4b

update FD and some nems configs

5f38b30

* all except lnd sbs run

Revert "update FD and some nems configs"

65eddac

This reverts commit 5f38b30.

junwang-noaa mentioned this pull request Oct 17, 2023

[INSTALL] fms/2023.02.01 JCSDA/spack-stack#823

Closed

DeniseWorthen and others added 10 commits October 17, 2023 13:55

change nems to ufs where needed

dab75e4

change a few print lines in FV3

95bae31

update CMEPS

bbcdbd0

Merge branch 'feature/nems2ufs' of https://github.com/DeniseWorthen/u…

4fed37d

…fs-weather-model into feature/nems2ufs

revert FV3 to develop

80b6767

* small print changes added to instlnd branch instead

Merge branch 'ufs-community:develop' into develop

453b567

Merge branch 'ufs-community:develop' into feature/nems2ufs

5075138

Update FV3

84795d2

Merge remote-tracking branch 'origin/develop' into add_zstandard

ee084d8

Merge branch 'ufs-community:develop' into develop

4da7e4f

Rename NEMS_CONFIGURE to UFS_CONFIGURE in three new cpld tests

630f983

BrianCurtis-NOAA and others added 6 commits October 31, 2023 16:12

Acorn RT Log

0c8c2d6

Merge branch 'add_zstandard' of github.com:DusanJovic-NOAA/ufs-weathe…

41bb687

…r-model into add_zstandard

WCOSS2 RT Log

2ae40cd

add orion RT logs: passed

00354c0

add jet RT log: passed

a5031d5

add hera RT log: passed

fb7dcdd

Revert .gitmodules and update FV3 and CMEPS

b150dcb

zach1221 requested review from BrianCurtis-NOAA, jkbk2004 and sadeghitabas October 31, 2023 20:32

jkbk2004 approved these changes Oct 31, 2023

View reviewed changes

sadeghitabas approved these changes Oct 31, 2023

View reviewed changes

jkbk2004 merged commit ce6c046 into ufs-community:develop Oct 31, 2023

jkbk2004 mentioned this pull request Oct 31, 2023

change references from "nems" to "ufs", allow file clobbering for GOCART history files #1948

Closed

42 tasks

DusanJovic-NOAA deleted the add_zstandard branch November 1, 2023 12:25

grantfirl mentioned this pull request Feb 2, 2024

UFS-dev PR#75 NCAR/ufs-weather-model#115

Merged

Conversation

DusanJovic-NOAA commented Oct 4, 2023 • edited by BrianCurtis-NOAA Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Author Checklist:

Description

Linked Issues and Pull Requests

Associated UFSWM Issue to close

Subcomponent Pull Requests

Blocking Dependencies

Subcomponents involved:

Anticipated Changes

Input data

Regression Tests:

Libraries

Testing Log:

Uh oh!

DusanJovic-NOAA commented Oct 5, 2023 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

DusanJovic-NOAA commented Oct 5, 2023

Uh oh!

DusanJovic-NOAA commented Oct 5, 2023

Uh oh!

DusanJovic-NOAA commented Oct 31, 2023

Uh oh!

DusanJovic-NOAA commented Oct 31, 2023

Uh oh!

DeniseWorthen commented Oct 31, 2023

Uh oh!

DeniseWorthen commented Oct 31, 2023

Uh oh!

DusanJovic-NOAA commented Oct 31, 2023

Uh oh!

BrianCurtis-NOAA commented Oct 31, 2023

Uh oh!

DeniseWorthen commented Oct 31, 2023

Uh oh!

zach1221 commented Oct 31, 2023

Uh oh!

DeniseWorthen commented Oct 31, 2023

Uh oh!

zach1221 commented Oct 31, 2023

Uh oh!

junwang-noaa commented Oct 31, 2023

Uh oh!

bbakernoaa commented Oct 31, 2023 via email

Uh oh!

DeniseWorthen commented Oct 31, 2023

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

9 participants

DusanJovic-NOAA commented Oct 4, 2023 •

edited by BrianCurtis-NOAA

Loading

DusanJovic-NOAA commented Oct 5, 2023 •

edited

Loading