Skip to content

Do Not Merge: Test ufs-utils from PR 815#1818

Closed
aerorahul wants to merge 1 commit into
NOAA-EMC:developfrom
aerorahul:feature/issue-1775
Closed

Do Not Merge: Test ufs-utils from PR 815#1818
aerorahul wants to merge 1 commit into
NOAA-EMC:developfrom
aerorahul:feature/issue-1775

Conversation

@aerorahul
Copy link
Copy Markdown
Contributor

@aerorahul aerorahul commented Aug 22, 2023

Description

This is a draft PR to test ufs-community/UFS_UTILS#815

Addresses global-workflow issue: #1775

DO NOT MERGE

@aerorahul aerorahul added CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Aug 22, 2023
@github-actions
Copy link
Copy Markdown

github-actions Bot commented Aug 22, 2023

Link to ReadTheDocs sample build for this PR can be found at:
https://global-workflow--1818.org.readthedocs.build/en/1818

@emcbot emcbot added CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion labels Aug 22, 2023
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera and removed CI-Hera-Ready **CM use only** PR is ready for CI testing on Hera labels Aug 22, 2023
@aerorahul aerorahul mentioned this pull request Aug 22, 2023
Merged
6 tasks
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress and removed CI-Hera-Building **Bot use only** CI testing is cloning/building on Hera labels Aug 22, 2023
@TerrenceMcGuinness-NOAA
Copy link
Copy Markdown
Collaborator

TerrenceMcGuinness-NOAA commented Aug 22, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Aug 22 18:44:23 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 18:48:27 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 19:12:30 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:33 UTC 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:36 UTC 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:39 UTC 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:45 UTC 2023 for experiment C96C48_hybatmDA_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:48 UTC 2023 for experiment C96_atm3DVar_2974811c

@emcbot emcbot added CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Aug 22, 2023
@emcbot
Copy link
Copy Markdown

emcbot commented Aug 22, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Tue Aug 22 13:44:22 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 13:46:43 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 14:16:31 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:37 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:40 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:43 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:45 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:49 CDT 2023 for experiment C96C48_hybatmDA_2974811c

@TerrenceMcGuinness-NOAA
Copy link
Copy Markdown
Collaborator

TerrenceMcGuinness-NOAA commented Aug 22, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Aug 22 18:44:23 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 18:48:27 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 19:12:30 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:33 UTC 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:36 UTC 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:39 UTC 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:45 UTC 2023 for experiment C96C48_hybatmDA_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:48 UTC 2023 for experiment C96_atm3DVar_2974811c
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Tue Aug 22 19:51:14 UTC 2023
with 3 successfully completed jobs

@emcbot
Copy link
Copy Markdown

emcbot commented Aug 22, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Tue Aug 22 13:44:22 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 13:46:43 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 14:16:31 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:37 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:40 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:43 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:45 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:49 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Tue Aug 22 15:00:34 CDT 2023
with 18 successfully completed jobs

@TerrenceMcGuinness-NOAA
Copy link
Copy Markdown
Collaborator

TerrenceMcGuinness-NOAA commented Aug 22, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Aug 22 18:44:23 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 18:48:27 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 19:12:30 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:33 UTC 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:36 UTC 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:39 UTC 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:45 UTC 2023 for experiment C96C48_hybatmDA_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:48 UTC 2023 for experiment C96_atm3DVar_2974811c
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Tue Aug 22 19:51:14 UTC 2023
with 3 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Tue Aug 22 20:27:08 UTC 2023
with 48 successfully completed jobs

@TerrenceMcGuinness-NOAA
Copy link
Copy Markdown
Collaborator

TerrenceMcGuinness-NOAA commented Aug 22, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Aug 22 18:44:23 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 18:48:27 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 19:12:30 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:33 UTC 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:36 UTC 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:39 UTC 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:45 UTC 2023 for experiment C96C48_hybatmDA_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:48 UTC 2023 for experiment C96_atm3DVar_2974811c
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Tue Aug 22 19:51:14 UTC 2023
with 3 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Tue Aug 22 20:27:08 UTC 2023
with 48 successfully completed jobs
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Tue Aug 22 20:36:14 UTC 2023
with 18 successfully completed jobs

@emcbot
Copy link
Copy Markdown

emcbot commented Aug 22, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Tue Aug 22 13:44:22 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 13:46:43 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 14:16:31 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:37 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:40 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:43 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:45 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:49 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Tue Aug 22 15:00:34 CDT 2023
with 18 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Tue Aug 22 15:36:20 CDT 2023
with 48 successfully completed jobs

@emcbot
Copy link
Copy Markdown

emcbot commented Aug 22, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Tue Aug 22 13:44:22 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 13:46:43 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 14:16:31 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:37 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:40 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:43 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:45 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:49 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Tue Aug 22 15:00:34 CDT 2023
with 18 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Tue Aug 22 15:36:20 CDT 2023
with 48 successfully completed jobs
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Tue Aug 22 15:52:09 CDT 2023
with 3 successfully completed jobs

@emcbot emcbot added CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Aug 22, 2023
@emcbot
Copy link
Copy Markdown

emcbot commented Aug 22, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Tue Aug 22 13:44:22 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 13:46:43 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 14:16:31 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:37 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:40 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:43 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:45 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 14:16:49 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Tue Aug 22 15:00:34 CDT 2023
with 18 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Tue Aug 22 15:36:20 CDT 2023
with 48 successfully completed jobs
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Tue Aug 22 15:52:09 CDT 2023
with 3 successfully completed jobs
Experiment C96_atm3DVar_2974811c Terminated: *FAILED*
Experiment C96_atm3DVar_2974811c Terminated with 2 tasks failed at Tue Aug 22 16:16:08 CDT 2023
Error logs:
/work2/noaa/stmp/GFS_CI_ROOT/PR/1818/RUNTESTS/COMROT/C96_atm3DVar_2974811c/logs/2021122100/gdasanal.log
/work2/noaa/stmp/GFS_CI_ROOT/PR/1818/RUNTESTS/COMROT/C96_atm3DVar_2974811c/logs/2021122100/gfsanal.log

@aerorahul
Copy link
Copy Markdown
Contributor Author

aerorahul commented Aug 22, 2023

@CatherineThomas-NOAA @GeorgeGayno-NOAA
Running with PR 815 from ufs-utils resulted in a failure in the GSI analysis jobs on Orion for both the GDAS and GFS cycles.
I don't see an obvious error message and am suspecting a glitch on Orion.
I will let the tests on Hera continue to see if they complete successfully.

@aerorahul
Copy link
Copy Markdown
Contributor Author

aerorahul commented Aug 22, 2023

@CatherineThomas-NOAA @GeorgeGayno-NOAA
I checked the tests on Hera and they do seem to be progressing without errors with the UFS-utils from PR 815. I am sure now the failures on Orion in the GSI analysis jobs were a result of something with the system. Looking at the timestamps on those failed logs, they appear to be at the same time, likely confirming my suspicion.

@TerrenceMcGuinness-NOAA
Copy link
Copy Markdown
Collaborator

TerrenceMcGuinness-NOAA commented Aug 22, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Aug 22 18:44:23 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 18:48:27 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 19:12:30 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:33 UTC 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:36 UTC 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:39 UTC 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:45 UTC 2023 for experiment C96C48_hybatmDA_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:48 UTC 2023 for experiment C96_atm3DVar_2974811c
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Tue Aug 22 19:51:14 UTC 2023
with 3 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Tue Aug 22 20:27:08 UTC 2023
with 48 successfully completed jobs
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Tue Aug 22 20:36:14 UTC 2023
with 18 successfully completed jobs
Experiment C96C48_hybatmDA_2974811c completed: *SUCCESS*
Experiment C96C48_hybatmDA_2974811c Completed at Tue Aug 22 22:51:10 UTC 2023
with 151 successfully completed jobs

@TerrenceMcGuinness-NOAA
Copy link
Copy Markdown
Collaborator

TerrenceMcGuinness-NOAA commented Aug 22, 2023 •
edited
Loading

Yes, I can see those jobs were CANCELED by Slurm. Not sure what the root cause it yet. Could have been done by the admins for up-and-coming maintenance or for account allocations or limits. I'll keep you guys posted if I can find out.

@TerrenceMcGuinness-NOAA
Copy link
Copy Markdown
Collaborator

TerrenceMcGuinness-NOAA commented Aug 22, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Aug 22 18:44:23 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 18:48:27 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 19:12:30 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:33 UTC 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:36 UTC 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:39 UTC 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:45 UTC 2023 for experiment C96C48_hybatmDA_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:48 UTC 2023 for experiment C96_atm3DVar_2974811c
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Tue Aug 22 19:51:14 UTC 2023
with 3 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Tue Aug 22 20:27:08 UTC 2023
with 48 successfully completed jobs
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Tue Aug 22 20:36:14 UTC 2023
with 18 successfully completed jobs
Experiment C96C48_hybatmDA_2974811c completed: *SUCCESS*
Experiment C96C48_hybatmDA_2974811c Completed at Tue Aug 22 22:51:10 UTC 2023
with 151 successfully completed jobs
Experiment C96_atm3DVar_2974811c completed: *SUCCESS*
Experiment C96_atm3DVar_2974811c Completed at Tue Aug 22 23:03:11 UTC 2023
with 89 successfully completed jobs

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added the CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully label Aug 22, 2023
@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA removed the CI-Hera-Running **Bot use only** CI testing on Hera for this PR is in-progress label Aug 22, 2023
@TerrenceMcGuinness-NOAA
Copy link
Copy Markdown
Collaborator

TerrenceMcGuinness-NOAA commented Aug 22, 2023

Automated global-workflow Testing Results:

Machine: Hera
Start: Tue Aug 22 18:44:23 UTC 2023 on hfe05
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Tue Aug 22 18:48:27 UTC 2023
Build:                         *SUCCESS*
Build: Completed at Tue Aug 22 19:12:30 UTC 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:33 UTC 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:36 UTC 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:39 UTC 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:45 UTC 2023 for experiment C96C48_hybatmDA_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Tue Aug 22 19:12:48 UTC 2023 for experiment C96_atm3DVar_2974811c
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Tue Aug 22 19:51:14 UTC 2023
with 3 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Tue Aug 22 20:27:08 UTC 2023
with 48 successfully completed jobs
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Tue Aug 22 20:36:14 UTC 2023
with 18 successfully completed jobs
Experiment C96C48_hybatmDA_2974811c completed: *SUCCESS*
Experiment C96C48_hybatmDA_2974811c Completed at Tue Aug 22 22:51:10 UTC 2023
with 151 successfully completed jobs
Experiment C96_atm3DVar_2974811c completed: *SUCCESS*
Experiment C96_atm3DVar_2974811c Completed at Tue Aug 22 23:03:11 UTC 2023
with 89 successfully completed jobs

@TerrenceMcGuinness-NOAA
Copy link
Copy Markdown
Collaborator

TerrenceMcGuinness-NOAA commented Aug 22, 2023 •
edited
Loading

Is this expected ...

mterry (Orion-login-1) 2021122100 $ grep FAILURE gdasanal.log | wc
571178 5140602 38268926
mterry (Orion-login-1) 2021122100 $ grep WARNING gdasanal.log | wc
97 684 7870
mterry (Orion-login-1) 2021122100 $ awk 'NR >= 575595 && NR <= 575600' gdasanal.log
73: Apply_AntCorr(FAILURE) : Input iFOV inconsistent with AC data
73: Apply_AntCorr(FAILURE) : Input iFOV inconsistent with AC data
73: Apply_AntCorr(FAILURE) : Input iFOV inconsistent with AC data
75: READ_PREPBUFR: WARNING invalid pressure pob= 0.000000000000000E+000
75: at k= 1 for obstype=t kx= 180 c_station_id=
75: 0022958 reset pob= 1000000000.00000
mterry (Orion-login-1) 2021122100 $

@aerorahul aerorahul mentioned this pull request Aug 23, 2023
Closed
@aerorahul
Copy link
Copy Markdown
Contributor Author

aerorahul commented Aug 23, 2023

Is this expected ...

mterry (Orion-login-1) 2021122100 $ grep FAILURE gdasanal.log | wc
571178 5140602 38268926
mterry (Orion-login-1) 2021122100 $ grep WARNING gdasanal.log | wc
97 684 7870
mterry (Orion-login-1) 2021122100 $ awk 'NR >= 575595 && NR <= 575600' gdasanal.log
73: Apply_AntCorr(FAILURE) : Input iFOV inconsistent with AC data
73: Apply_AntCorr(FAILURE) : Input iFOV inconsistent with AC data
73: Apply_AntCorr(FAILURE) : Input iFOV inconsistent with AC data
75: READ_PREPBUFR: WARNING invalid pressure pob= 0.000000000000000E+000
75: at k= 1 for obstype=t kx= 180 c_station_id=
75: 0022958 reset pob= 1000000000.00000
mterry (Orion-login-1) 2021122100 $

I am not sure @TerrenceMcGuinness-NOAA
This is for the science team to verify if this output is expected. Either way, the runs on Hera completed without an error, so I am confident the Orion failures were machine related and these may not be relevant to the failure.

@CatherineThomas-NOAA
Copy link
Copy Markdown
Contributor

CatherineThomas-NOAA commented Aug 23, 2023

Thanks for setting up these tests @aerorahul. Can you or @TerrenceMcGuinness-NOAA point me to the full log for the Orion failure? I don't think it's related to this PR but I'd like to check out the antenna correction error.

@aerorahul
Copy link
Copy Markdown
Contributor Author

aerorahul commented Aug 24, 2023

Failed logs on Orion at:
/work2/noaa/stmp/GFS_CI_ROOT/PR/1818/RUNTESTS/COMROT/C96_atm3DVar_2974811c/logs/2021122100/gdasanal.log
/work2/noaa/stmp/GFS_CI_ROOT/PR/1818/RUNTESTS/COMROT/C96_atm3DVar_2974811c/logs/2021122100/gfsanal.log

@TerrenceMcGuinness-NOAA
Copy link
Copy Markdown
Collaborator

TerrenceMcGuinness-NOAA commented Aug 25, 2023

Let's set off the Orion CI tests again since it's been a full day after the maintenance updates.

@TerrenceMcGuinness-NOAA TerrenceMcGuinness-NOAA added CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion and removed CI-Orion-Failed **Bot use only** CI testing on Orion for this PR has failed labels Aug 25, 2023
@emcbot emcbot added CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress and removed CI-Orion-Ready **CM use only** PR is ready for CI testing on Orion CI-Orion-Building **Bot use only** CI testing is cloning/building on Orion labels Aug 25, 2023
@emcbot
Copy link
Copy Markdown

emcbot commented Aug 25, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Aug 25 08:48:38 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug 25 08:51:07 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug 25 09:19:48 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:19:57 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:01 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:04 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:08 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:13 CDT 2023 for experiment C96C48_hybatmDA_2974811c

@emcbot
Copy link
Copy Markdown

emcbot commented Aug 25, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Aug 25 08:48:38 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug 25 08:51:07 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug 25 09:19:48 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:19:57 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:01 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:04 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:08 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:13 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Fri Aug 25 10:00:48 CDT 2023
with 18 successfully completed jobs

@emcbot
Copy link
Copy Markdown

emcbot commented Aug 25, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Aug 25 08:48:38 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug 25 08:51:07 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug 25 09:19:48 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:19:57 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:01 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:04 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:08 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:13 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Fri Aug 25 10:00:48 CDT 2023
with 18 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Fri Aug 25 10:32:09 CDT 2023
with 48 successfully completed jobs

@emcbot
Copy link
Copy Markdown

emcbot commented Aug 25, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Aug 25 08:48:38 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug 25 08:51:07 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug 25 09:19:48 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:19:57 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:01 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:04 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:08 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:13 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Fri Aug 25 10:00:48 CDT 2023
with 18 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Fri Aug 25 10:32:09 CDT 2023
with 48 successfully completed jobs
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Fri Aug 25 10:36:17 CDT 2023
with 3 successfully completed jobs

@emcbot
Copy link
Copy Markdown

emcbot commented Aug 25, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Aug 25 08:48:38 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug 25 08:51:07 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug 25 09:19:48 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:19:57 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:01 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:04 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:08 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:13 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Fri Aug 25 10:00:48 CDT 2023
with 18 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Fri Aug 25 10:32:09 CDT 2023
with 48 successfully completed jobs
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Fri Aug 25 10:36:17 CDT 2023
with 3 successfully completed jobs
Experiment C96_atm3DVar_2974811c completed: *SUCCESS*
Experiment C96_atm3DVar_2974811c Completed at Fri Aug 25 12:56:08 CDT 2023
with 89 successfully completed jobs

@emcbot
Copy link
Copy Markdown

emcbot commented Aug 25, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Aug 25 08:48:38 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug 25 08:51:07 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug 25 09:19:48 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:19:57 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:01 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:04 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:08 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:13 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Fri Aug 25 10:00:48 CDT 2023
with 18 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Fri Aug 25 10:32:09 CDT 2023
with 48 successfully completed jobs
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Fri Aug 25 10:36:17 CDT 2023
with 3 successfully completed jobs
Experiment C96_atm3DVar_2974811c completed: *SUCCESS*
Experiment C96_atm3DVar_2974811c Completed at Fri Aug 25 12:56:08 CDT 2023
with 89 successfully completed jobs
Experiment C96C48_hybatmDA_2974811c completed: *SUCCESS*
Experiment C96C48_hybatmDA_2974811c Completed at Fri Aug 25 13:06:09 CDT 2023
with 151 successfully completed jobs

@emcbot emcbot added CI-Orion-Passed **Bot use only** CI testing on Orion for this PR has completed successfully and removed CI-Orion-Running **Bot use only** CI testing on Orion for this PR is in-progress labels Aug 25, 2023
@emcbot
Copy link
Copy Markdown

emcbot commented Aug 25, 2023

Automated global-workflow Testing Results:

Machine: Orion
Start: Fri Aug 25 08:48:38 CDT 2023 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Checkout:                      *SUCCESS*
Checkout: Completed at Fri Aug 25 08:51:07 CDT 2023
Build:                         *SUCCESS*
Build: Completed at Fri Aug 25 09:19:48 CDT 2023
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:19:57 CDT 2023 for experiment C48_ATM_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:01 CDT 2023 for experiment C48_S2SA_gefs_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:04 CDT 2023 for experiment C48_S2S_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:08 CDT 2023 for experiment C96_atm3DVar_2974811c
Created experiment:            *SUCCESS*
Case setup: Completed at Fri Aug 25 09:20:13 CDT 2023 for experiment C96C48_hybatmDA_2974811c
Experiment C48_S2S_2974811c completed: *SUCCESS*
Experiment C48_S2S_2974811c Completed at Fri Aug 25 10:00:48 CDT 2023
with 18 successfully completed jobs
Experiment C48_ATM_2974811c completed: *SUCCESS*
Experiment C48_ATM_2974811c Completed at Fri Aug 25 10:32:09 CDT 2023
with 48 successfully completed jobs
Experiment C48_S2SA_gefs_2974811c completed: *SUCCESS*
Experiment C48_S2SA_gefs_2974811c Completed at Fri Aug 25 10:36:17 CDT 2023
with 3 successfully completed jobs
Experiment C96_atm3DVar_2974811c completed: *SUCCESS*
Experiment C96_atm3DVar_2974811c Completed at Fri Aug 25 12:56:08 CDT 2023
with 89 successfully completed jobs
Experiment C96C48_hybatmDA_2974811c completed: *SUCCESS*
Experiment C96C48_hybatmDA_2974811c Completed at Fri Aug 25 13:06:09 CDT 2023
with 151 successfully completed jobs

@aerorahul
Copy link
Copy Markdown
Contributor Author

aerorahul commented Aug 28, 2023

Closing this PR as testing has concluded.

@aerorahul aerorahul closed this Aug 28, 2023
@CatherineThomas-NOAA
Copy link
Copy Markdown
Contributor

CatherineThomas-NOAA commented Aug 31, 2023

73: Apply_AntCorr(FAILURE) : Input iFOV inconsistent with AC data

These errors in the Orion log are from a problem with the CRTM installation. As pointed out by @emilyhcliu, the MHS MetOp-C coefficient file is incorrect. It only contains header information:

Orion-login-4[4] cthomas$ pwd
/apps/contrib/NCEP/libs/hpc-stack/intel-2018.4/crtm/2.4.0/fix
Orion-login-4[5] cthomas$ ls -l mhs*
-rw-r----- 1 8536 noaa-hpc 11396 Feb  4  2022 mhs_metop-a.SpcCoeff.bin
-rw-r----- 1 8536 noaa-hpc 49212 Feb  4  2022 mhs_metop-a.TauCoeff.bin
-rw-r----- 1 8536 noaa-hpc 11396 Feb  4  2022 mhs_metop-b.SpcCoeff.bin
-rw-r----- 1 8536 noaa-hpc 49212 Feb  4  2022 mhs_metop-b.TauCoeff.bin
-rw-r----- 1 8536 noaa-hpc   492 Feb  4  2022 mhs_metop-c.SpcCoeff.bin
-rw-r----- 1 8536 noaa-hpc 49212 Feb  4  2022 mhs_metop-c.TauCoeff.bin

Which repo should I open an issue under to get this fixed?

@aerorahul
Copy link
Copy Markdown
Contributor Author

aerorahul commented Aug 31, 2023

seems like an installation issue with CRTM.
hpc-stack seems relevant.

@CatherineThomas-NOAA CatherineThomas-NOAA mentioned this pull request Aug 31, 2023
Open
@aerorahul aerorahul deleted the feature/issue-1775 branch December 3, 2023 16:54
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@DavidHuber-NOAA DavidHuber-NOAA Awaiting requested review from DavidHuber-NOAA DavidHuber-NOAA will be requested when the pull request is marked ready for review DavidHuber-NOAA is a code owner

@TravisElless-NOAA TravisElless-NOAA Awaiting requested review from TravisElless-NOAA TravisElless-NOAA will be requested when the pull request is marked ready for review TravisElless-NOAA is a code owner

Assignees

No one assigned

Labels

CI-Hera-Passed **Bot use only** CI testing on Hera for this PR has completed successfully CI-Orion-Passed **Bot use only** CI testing on Orion for this PR has completed successfully

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

4 participants

@aerorahul @TerrenceMcGuinness-NOAA @emcbot @CatherineThomas-NOAA