Skip to content

[DO NOT MERGE] Weekly CI Tests Friday Jan 19, 2024#2240

Closed
emcbot wants to merge 1 commit into
NOAA-EMC:developfrom
emcbot:weekly_ci_20240119
Closed

[DO NOT MERGE] Weekly CI Tests Friday Jan 19, 2024#2240
emcbot wants to merge 1 commit into
NOAA-EMC:developfrom
emcbot:weekly_ci_20240119

Conversation

@emcbot
Copy link
Copy Markdown

@emcbot emcbot commented Jan 19, 2024

[DO NOT MERGE] Weekly CI Tests Friday Jan 19, 2024

@emcbot emcbot marked this pull request as draft January 19, 2024 22:47
@emcbot emcbot added CI/CD Issue related to CI/CD CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Jan 19, 2024
@emcbot
Copy link
Copy Markdown
Author

emcbot commented Jan 19, 2024 

CI Update on Orion at 01/19/24 04:48:08 PM
============================================
Cloning and Building global-workflow PR: 2240
with PID: 58721 on host: Orion-login-1

@emcbot emcbot added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Jan 19, 2024
@emcbot
Copy link
Copy Markdown
Author

emcbot commented Jan 19, 2024 

CI Update on Hera at 01/19/24 10:48:11 PM
============================================
Cloning and Building global-workflow PR: 2240
with PID: 2097 on host: hfe05

@emcbot emcbot added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Jan 19, 2024
@emcbot
Copy link
Copy Markdown
Author

emcbot commented Jan 19, 2024

Automated global-workflow Testing Results:

Machine: Hera
Start: Fri Jan 19 23:02:08 UTC 2024 on hfe05
---------------------------------------------------
Build: Completed at 01/19/24 11:46:48 PM
Case setup: Completed for experiment C384C192_hybatmda_b0ee609e
Case setup: Completed for experiment C384_S2SWA_b0ee609e
Case setup: Completed for experiment C384_atm3DVar_b0ee609e

@emcbot emcbot added CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Jan 19, 2024
@emcbot
Copy link
Copy Markdown
Author

emcbot commented Jan 19, 2024

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Jan 19 16:51:49 CST 2024 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build: Completed at 01/19/24 05:50:04 PM
Case setup: Completed for experiment C384_atm3DVar_b0ee609e
Case setup: Completed for experiment C384C192_hybatmda_b0ee609e
Case setup: Completed for experiment C384_S2SWA_b0ee609e

@emcbot emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Jan 20, 2024
@emcbot
Copy link
Copy Markdown
Author

emcbot commented Jan 20, 2024 

Experiment C384_S2SWA_b0ee609e  *** FAILED *** on Orion
Experiment C384_S2SWA_b0ee609e  with 1 tasks failed at 01/19/24 06:20:32 PM
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/ORION/PR/2240/RUNTESTS/COMROOT/C384_S2SWA_b0ee609e/logs/2016070100/gfsfcst.log

@emcbot emcbot added CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed and removed CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress labels Jan 20, 2024
@emcbot
Copy link
Copy Markdown
Author

emcbot commented Jan 20, 2024 

Experiment C384_S2SWA_b0ee609e  *** FAILED *** on Hera
Experiment C384_S2SWA_b0ee609e  with 1 tasks failed at 01/20/24 12:32:22 AM
Error logs:
/scratch1/NCEPDEV/global/Terry.McGuinness/GFS_CI_ROOT/PR/2240/RUNTESTS/COMROOT/C384_S2SWA_b0ee609e/logs/2016070100/gfsfcst.log

@DavidHuber-NOAA
Copy link
Copy Markdown
Contributor

DavidHuber-NOAA commented Jan 22, 2024

The C384_S2SWA GFS forecast is failing on both Orion and Hera with MAPL errors starting in ExtDataGridCompMod.F90:

   0:  in radiation_clouds_prop=           8 F           4 F F           2           1
   0:  in radiation_clouds_prop=           8 F           4 F F           2           1
   0:  in radiation_clouds_prop=           8 F           4 F F           2           1
   0:  in radiation_clouds_prop=           8 F           4 F F           2           1
   0:  in radiation_clouds_prop=           8 F           4 F F           2           1
 762: pe=00762 FAIL at line=02588    ExtDataGridCompMod.F90                   <unknown error>
 762: pe=00762 FAIL at line=01323    ExtDataGridCompMod.F90                   <status=-1>
 762: pe=00762 FAIL at line=01901    MAPL_Generic.F90                         <status=-1>
 762: pe=00762 FAIL at line=01243    MAPL_CapGridComp.F90                     <status=-1>
 762: pe=00762 FAIL at line=01206    MAPL_CapGridComp.F90                     <status=-1>
 762: pe=00762 FAIL at line=01166    MAPL_CapGridComp.F90                     <status=-1>
 762: pe=00762 FAIL at line=00834    MAPL_CapGridComp.F90                     <status=-1>
 762: pe=00762 FAIL at line=00974    MAPL_CapGridComp.F90                     <status=-1>
...

@WalterKolczynski-NOAA
Copy link
Copy Markdown
Contributor

WalterKolczynski-NOAA commented Jan 22, 2024

May be related to the changes last week in #2201, though the low-res tests passed. @bbakernoaa can you take a look?

@bbakernoaa
Copy link
Copy Markdown
Contributor

bbakernoaa commented Jan 22, 2024

Yea I'll take a look first thing tomorrow

@bbakernoaa
Copy link
Copy Markdown
Contributor

bbakernoaa commented Jan 23, 2024

@WalterKolczynski-NOAA @DavidHuber-NOAA

Found the issue

We needed to switch the flag for climatology in the biogenic files. Do you want me to setup a separate PR or combine it with this one?

The change is very easy. The lines that needs changed are here: https://github.com/NOAA-EMC/global-workflow/blob/develop/parm/ufs/gocart/ExtData.other#L57-L60

specifically

OC_ISOPRENE NA  N Y %y4-%m2-%d2t12:00:00 none none isoprene ExtData/nexus/MEGAN_OFFLINE_BVOC/v2021-12/MEGAN_OFFLINE_CLIMO_2000_2022_%m2.nc
OC_LIMO     NA  N Y %y4-%m2-%d2t12:00:00 none none limo     ExtData/nexus/MEGAN_OFFLINE_BVOC/v2021-12/MEGAN_OFFLINE_CLIMO_2000_2022_%m2.nc
OC_MTPA     NA  N Y %y4-%m2-%d2t12:00:00 none none mtpa     ExtData/nexus/MEGAN_OFFLINE_BVOC/v2021-12/MEGAN_OFFLINE_CLIMO_2000_2022_%m2.nc
OC_MTPO     NA  N Y %y4-%m2-%d2t12:00:00 none none mtpo     ExtData/nexus/MEGAN_OFFLINE_BVOC/v2021-12/MEGAN_OFFLINE_CLIMO_2000_2022_%m2.nc

needs to be changed to Y in the 3rd column. Like so....

OC_ISOPRENE NA  Y Y %y4-%m2-%d2t12:00:00 none none isoprene ExtData/nexus/MEGAN_OFFLINE_BVOC/v2021-12/MEGAN_OFFLINE_CLIMO_2000_2022_%m2.nc
OC_LIMO     NA  Y Y %y4-%m2-%d2t12:00:00 none none limo     ExtData/nexus/MEGAN_OFFLINE_BVOC/v2021-12/MEGAN_OFFLINE_CLIMO_2000_2022_%m2.nc
OC_MTPA     NA  Y Y %y4-%m2-%d2t12:00:00 none none mtpa     ExtData/nexus/MEGAN_OFFLINE_BVOC/v2021-12/MEGAN_OFFLINE_CLIMO_2000_2022_%m2.nc
OC_MTPO     NA  Y Y %y4-%m2-%d2t12:00:00 none none mtpo     ExtData/nexus/MEGAN_OFFLINE_BVOC/v2021-12/MEGAN_OFFLINE_CLIMO_2000_2022_%m2.nc

@WalterKolczynski-NOAA
Copy link
Copy Markdown
Contributor

WalterKolczynski-NOAA commented Jan 23, 2024

@bbakernoaa Great job. Separate PR please, this "PR" isn't a real PR, it is a kludge for the CI system to run weekly testing.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DavidHuber-NOAA DavidHuber-NOAA Awaiting requested review from DavidHuber-NOAA DavidHuber-NOAA will be requested when the pull request is marked ready for review DavidHuber-NOAA is a code owner

@TravisElless-NOAA TravisElless-NOAA Awaiting requested review from TravisElless-NOAA TravisElless-NOAA will be requested when the pull request is marked ready for review TravisElless-NOAA is a code owner

Assignees

No one assigned

Labels

CI/CD Issue related to CI/CD CI-Hera-Failed **Bot use only** CI testing on Hera for this PR has failed CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

5 participants

@emcbot @DavidHuber-NOAA @WalterKolczynski-NOAA @bbakernoaa @TerrenceMcGuinness-NOAA