CI self-test with KEEPDATA=YES#2734
Conversation
TerrenceMcGuinness-NOAA
added
CI-Hera-Ready
emcbot
added
CI-Hera-Building
|
Experiment C96_atmaerosnowDA FAILED on Hera with error logs: Follow link here to view the contents of the above file(s): (link) |
emcbot
added
CI-Hera-Failed
|
Experiment C96_atmaerosnowDA FAILED on Hera in |
|
Experiment C48mx500_3DVarAOWCDA FAILED on Hera in |
|
C48mx500_3DVarAOWCDA failure The C48mx500_3DVarAOWCDA failure in this PR is the same as #2700. The 20210324 18Z gdasfcst aborts As @guillaumevernieres notes, the log file contains This PR uses an older |
|
C96_atmaerosnowDA failure The C96_atmaerosnowDA failure in this PR differs from PR #2700 and #2729. The 20211220 18Z gdassfcanl fails in this PR with the error message Local file According to gdassnowanl.log file File I am not familiar with snow DA. Tagging @jiaruidong2017 . Jiarui, what are your thoughts on this failure? |
|
Thanks @RussTreadon-NOAA for digging this. I actually don't have any idea why this happened, and I didn't meet such an issue from my previous tests. A rerun to this CI test may help to find the reason. @CoryMartin-NOAA do you have any thoughts on this? |
|
Thank you @jiaruidong2017 for your reply. Do you routinely run C96_atmaerosnowDA as part of your development? If not, how do / how frequently do you test JEDI snow DA in g-w? |
|
@RussTreadon-NOAA I actually didn't run the C96_atmaerosnowDA CI test for my development work, but instead I run my own JEDI snow DA test. Recently, I have run my tests four times over the past two weeks. |
|
@jiaruidong2017 , to help with debugging, when did you make these runs, on which machine, and do you still have the log files online? |
|
@RussTreadon-NOAA You can find the following log files for my three tests as: /scratch1/NCEPDEV/climate/Jiarui.Dong/ptmp/cory04/logs/ (Today) |
@JessicaMeixner-NOAA just checked, the ocean and seaice increments are all nans. |
|
PR #2681 was not tested on Hera. I'm not sure why it was not (I know stmp was an issue, but this PR changes a lot for WCDA), but I think this could be the cause of the WCDA failures we are seeing and perhaps because of some logic clean up at the end or oversights in non-CI testing this was not seen. It also seems that #2719 is also possibly causing issues for tests not related to WCDA based on some other threads. |
@JessicaMeixner-NOAA |
TerrenceMcGuinness-NOAA
added
CI-Hera-Ready
emcbot
added
CI-Hera-Building
|
Experiment C48_ATM FAILED on Hera in |
|
Experiment C96_atm3DVar FAILED on Hera in |
|
Experiment C48mx500_3DVarAOWCDA FAILED on Hera with error logs: Follow link here to view the contents of the above file(s): (link) |
|
Experiment C96_atmaerosnowDA FAILED on Hera in |
|
Experiment C48_S2SW FAILED on Hera in |
|
Experiment C48mx500_3DVarAOWCDA FAILED on Hera in |
|
Experiment C96C48_hybatmDA FAILED on Hera in |
|
Experiment C48_S2SWA_gefs FAILED on Hera in |
TerrenceMcGuinness-NOAA
added
CI-Hera-Ready
emcbot
added
CI-Hera-Building
|
CI Passed Hera at |
|
Disk requirements for RUNDIRS with KEEPDATA=YES: 437 G And the requirements in all the EXPDIRs (in a typical CI run): 432 G |
|
@TerrenceMcGuinness-NOAA |
…NOAA-EMC#2734; Add option in GFS PBL to use liquid/ice potential temperature in local mixing NOAA-EMC#2763; Update MOM6 to its main repo. 20250527 commit NOAA-EMC#2757; Mom6 025 warmstart option NOAA-EMC#2761 (NOAA-EMC#2734) * UFSWM - change to MOM6 namelist file * FV3 - * ccpp-physics - Bring in the scale-aware 3DTKE related changes for GFS TKE EDMF PBL scheme * ccpp-physics - Add a namelist option in GFS TKE EDMF PBL scheme to apply an approximate way to represent local mixing/diffusion process in T-equation using liquid/ice water potential temperature. * atmos_cubed_sphere - Enable the scale-aware 3DTKE capability for the GFS TKE EDMF PBL scheme * MOM6 - update MOM6 to its main repo. 20250527 commit (originally GFDL 20250423 candidate) --------- Co-authored-by: jiandewang <jiande.wang@noaa.gov> Co-authored-by: NeilBarton-NOAA <neil.barton@noaa.gov>
7 participants
Description
This is a CI self-test with KEEPDATA=YES for save off of RUNDIRS to capture disk costs of running CI tests.