Skip to content

UFS-dev PR#25#79

Merged
grantfirl merged 12 commits into
NCAR:mainfrom
grantfirl:ufs-dev-PR25
Dec 20, 2022
Merged

UFS-dev PR#25#79
grantfirl merged 12 commits into
NCAR:mainfrom
grantfirl:ufs-dev-PR25

Conversation

@grantfirl
Copy link
Copy Markdown
Collaborator

Identical to ufs-community#1511

changed RT BLs for NoahMP tests

BrianCurtis-NOAA and others added 3 commits December 1, 2022 15:08
…ault (ufs-community#1503), WCOSS2 MAPL Enable Tests (ufs-community#1507) (ufs-community#1505)

* Inline post is now compiled by default on all supported platforms

* Create Debug RT for AQM, Bring in fixes from debug testing and @rmontuoro small updates to AQM

* update wcoss2 module files with esmf/mapl and turn on gocart in coupled runs

* disable regional_atmaq_debug on jet: time out issue

Co-authored-by: Dusan Jovic <dusan.jovic@noaa.gov>
Co-authored-by: Jun Wang <junwang-noaa@users.noreply.github.com>
Co-authored-by: jkbk2004 <jong.kim@noaa.gov>
* Change fv3atm branch to bugfix/noahmp_tq2m

Co-authored-by: jkbk2004 <jong.kim@noaa.gov>
Co-authored-by: Brian Curtis <brian.curtis@noaa.gov>
@grantfirl
Copy link
Copy Markdown
Collaborator Author

contains changes from previous PRs that will disappear once they are merged and this branch is updated

@dustinswales
Copy link
Copy Markdown
Member

Automated RT Failure Notification
Machine: cheyenne
Compiler: gnu
Job: RT
[RT] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1143720039/20221216151506/ufs-weather-model
[RT] Error: Test control_p8 006 failed in check_result failed
[RT] Error: Test control_p8 006 failed in run_test failed
[RT] Error: Test rrfs_v1beta 018 failed in check_result failed
[RT] Error: Test rrfs_v1beta 018 failed in run_test failed
[RT] Error: Test rrfs_v1beta_debug 031 failed in check_result failed
[RT] Error: Test rrfs_v1beta_debug 031 failed in run_test failed
[RT] Error: Test control_debug_p8 034 failed in check_result failed
[RT] Error: Test control_debug_p8 034 failed in run_test failed
[RT] Error: Test cpld_control_p8 049 failed in check_result failed
[RT] Error: Test cpld_control_p8 049 failed in run_test failed
[RT] Error: Test cpld_control_nowave_noaero_p8 050 failed in check_result failed
[RT] Error: Test cpld_control_nowave_noaero_p8 050 failed in run_test failed
[RT] Error: Test cpld_debug_p8 051 failed in check_result failed
[RT] Error: Test cpld_debug_p8 051 failed in run_test failed
Please make changes and add the following label back: cheyenne-gnu-RT

@dustinswales
Copy link
Copy Markdown
Member

Automated RT Failure Notification
Machine: hera
Compiler: gnu
Job: RT
[RT] Repo location: /scratch1/BMC/gmtb/RT/auto_RT/Pull_Requests/1143720039/20221216221519/ufs-weather-model
[RT] Error: Test control_p8 006 failed in check_result failed
[RT] Error: Test control_p8 006 failed in run_test failed
[RT] Error: Test rrfs_v1beta 018 failed in check_result failed
[RT] Error: Test rrfs_v1beta 018 failed in run_test failed
[RT] Error: Test rrfs_v1beta_debug 031 failed in check_result failed
[RT] Error: Test rrfs_v1beta_debug 031 failed in run_test failed
[RT] Error: Test control_debug_p8 034 failed in check_result failed
[RT] Error: Test control_debug_p8 034 failed in run_test failed
[RT] Error: Test cpld_control_p8 049 failed in run_test failed
[RT] Error: Test cpld_control_nowave_noaero_p8 050 failed in check_result failed
[RT] Error: Test cpld_control_nowave_noaero_p8 050 failed in run_test failed
[RT] Error: Test cpld_debug_p8 051 failed in check_result failed
[RT] Error: Test cpld_debug_p8 051 failed in run_test failed
Please make changes and add the following label back: hera-gnu-RT

@dustinswales
Copy link
Copy Markdown
Member

Automated RT Failure Notification
Machine: hera
Compiler: intel
Job: RT
[RT] Repo location: /scratch1/BMC/gmtb/RT/auto_RT/Pull_Requests/1143720039/20221216223014/ufs-weather-model
[RT] Error: Test cpld_control_p8 001 failed in check_result failed
[RT] Error: Test cpld_control_p8 001 failed in run_test failed
[RT] Error: Test cpld_2threads_p8 003 failed in check_result failed
[RT] Error: Test cpld_2threads_p8 003 failed in run_test failed
[RT] Error: Test cpld_esmfthreads_p8 004 failed in check_result failed
[RT] Error: Test cpld_esmfthreads_p8 004 failed in run_test failed
[RT] Error: Test cpld_decomp_p8 005 failed in check_result failed
[RT] Error: Test cpld_decomp_p8 005 failed in run_test failed
[RT] Error: Test cpld_mpi_p8 006 failed in check_result failed
[RT] Error: Test cpld_mpi_p8 006 failed in run_test failed
[RT] Error: Test cpld_control_ciceC_p8 007 failed in check_result failed
[RT] Error: Test cpld_control_ciceC_p8 007 failed in run_test failed
[RT] Error: Test cpld_control_c192_p8 008 failed in check_result failed
[RT] Error: Test cpld_control_c192_p8 008 failed in run_test failed
[RT] Error: Test cpld_bmark_p8 010 failed in check_result failed
[RT] Error: Test cpld_bmark_p8 010 failed in run_test failed
[RT] Error: Test cpld_control_noaero_p8 012 failed in check_result failed
[RT] Error: Test cpld_control_noaero_p8 012 failed in run_test failed
[RT] Error: Test cpld_control_nowave_noaero_p8 013 failed in check_result failed
[RT] Error: Test cpld_control_nowave_noaero_p8 013 failed in run_test failed
[RT] Error: Test cpld_debug_p8 014 failed in check_result failed
[RT] Error: Test cpld_debug_p8 014 failed in run_test failed
[RT] Error: Test cpld_debug_noaero_p8 015 failed in check_result failed
[RT] Error: Test cpld_debug_noaero_p8 015 failed in run_test failed
[RT] Error: Test cpld_control_noaero_p8_agrid 016 failed in check_result failed
[RT] Error: Test cpld_control_noaero_p8_agrid 016 failed in run_test failed
[RT] Error: Test cpld_control_c48 017 failed in check_result failed
[RT] Error: Test cpld_control_c48 017 failed in run_test failed
[RT] Error: Test cpld_warmstart_c48 018 failed in check_result failed
[RT] Error: Test cpld_warmstart_c48 018 failed in run_test failed
[RT] Error: Test control_p8 040 failed in check_result failed
[RT] Error: Test control_p8 040 failed in run_test failed
[RT] Error: Test control_p8_lndp 041 failed in check_result failed
[RT] Error: Test control_p8_lndp 041 failed in run_test failed
[RT] Error: Test control_decomp_p8 043 failed in check_result failed
[RT] Error: Test control_decomp_p8 043 failed in run_test failed
[RT] Error: Test control_2threads_p8 044 failed in check_result failed
[RT] Error: Test control_2threads_p8 044 failed in run_test failed
[RT] Error: Test control_p8_rrtmgp 045 failed in check_result failed
[RT] Error: Test control_p8_rrtmgp 045 failed in run_test failed
[RT] Error: Test merra2_thompson 046 failed in check_result failed
[RT] Error: Test merra2_thompson 046 failed in run_test failed
[RT] Error: Test rrfs_v1beta 068 failed in check_result failed
[RT] Error: Test rrfs_v1beta 068 failed in run_test failed
[RT] Error: Test rrfs_v1nssl 069 failed in check_result failed
[RT] Error: Test rrfs_v1nssl 069 failed in run_test failed
[RT] Error: Test rrfs_v1nssl_nohailnoccn 070 failed in check_result failed
[RT] Error: Test rrfs_v1nssl_nohailnoccn 070 failed in run_test failed
[RT] Error: Test control_debug_p8 092 failed in check_result failed
[RT] Error: Test control_debug_p8 092 failed in run_test failed
[RT] Error: Test rrfs_v1beta_debug 107 failed in check_result failed
[RT] Error: Test rrfs_v1beta_debug 107 failed in run_test failed
[RT] Error: Test control_p8_atmlnd_sbs 154 failed in check_result failed
[RT] Error: Test control_p8_atmlnd_sbs 154 failed in run_test failed
[RT] Error: Test atmaero_control_p8 156 failed in check_result failed
[RT] Error: Test atmaero_control_p8 156 failed in run_test failed
[RT] Error: Test atmaero_control_p8_rad 157 failed in check_result failed
[RT] Error: Test atmaero_control_p8_rad 157 failed in run_test failed
[RT] Error: Test atmaero_control_p8_rad_micro 158 failed in check_result failed
[RT] Error: Test atmaero_control_p8_rad_micro 158 failed in run_test failed
[RT] Error: Test regional_atmaq 159 failed in check_result failed
[RT] Error: Test regional_atmaq 159 failed in run_test failed
[RT] Error: Test regional_atmaq_debug 160 failed in check_result failed
[RT] Error: Test regional_atmaq_debug 160 failed in run_test failed
Please make changes and add the following label back: hera-intel-RT

@dustinswales
Copy link
Copy Markdown
Member

Automated RT Failure Notification
Machine: cheyenne
Compiler: intel
Job: RT
[RT] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1143720039/20221216153009/ufs-weather-model
[RT] Error: Test cpld_control_p8 001 failed in check_result failed
[RT] Error: Test cpld_control_p8 001 failed in run_test failed
[RT] Error: Test cpld_2threads_p8 003 failed in check_result failed
[RT] Error: Test cpld_2threads_p8 003 failed in run_test failed
[RT] Error: Test cpld_esmfthreads_p8 004 failed in check_result failed
[RT] Error: Test cpld_esmfthreads_p8 004 failed in run_test failed
[RT] Error: Test cpld_decomp_p8 005 failed in check_result failed
[RT] Error: Test cpld_decomp_p8 005 failed in run_test failed
[RT] Error: Test cpld_mpi_p8 006 failed in check_result failed
[RT] Error: Test cpld_mpi_p8 006 failed in run_test failed
[RT] Error: Test cpld_control_ciceC_p8 007 failed in check_result failed
[RT] Error: Test cpld_control_ciceC_p8 007 failed in run_test failed
[RT] Error: Test cpld_control_c192_p8 008 failed in check_result failed
[RT] Error: Test cpld_control_c192_p8 008 failed in run_test failed
[RT] Error: Test cpld_control_noaero_p8 010 failed in check_result failed
[RT] Error: Test cpld_control_noaero_p8 010 failed in run_test failed
[RT] Error: Test cpld_control_nowave_noaero_p8 011 failed in check_result failed
[RT] Error: Test cpld_control_nowave_noaero_p8 011 failed in run_test failed
[RT] Error: Test cpld_debug_p8 012 failed in check_result failed
[RT] Error: Test cpld_debug_p8 012 failed in run_test failed
[RT] Error: Test cpld_debug_noaero_p8 013 failed in check_result failed
[RT] Error: Test cpld_debug_noaero_p8 013 failed in run_test failed
[RT] Error: Test cpld_control_noaero_p8_agrid 014 failed in check_result failed
[RT] Error: Test cpld_control_noaero_p8_agrid 014 failed in run_test failed
[RT] Error: Test cpld_control_c48 015 failed in check_result failed
[RT] Error: Test cpld_control_c48 015 failed in run_test failed
[RT] Error: Test cpld_warmstart_c48 016 failed in check_result failed
[RT] Error: Test cpld_warmstart_c48 016 failed in run_test failed
[RT] Error: Test control_p8 038 failed in check_result failed
[RT] Error: Test control_p8 038 failed in run_test failed
[RT] Error: Test control_p8_lndp 039 failed in check_result failed
[RT] Error: Test control_p8_lndp 039 failed in run_test failed
[RT] Error: Test control_decomp_p8 041 failed in check_result failed
[RT] Error: Test control_decomp_p8 041 failed in run_test failed
[RT] Error: Test control_2threads_p8 042 failed in check_result failed
[RT] Error: Test control_2threads_p8 042 failed in run_test failed
[RT] Error: Test control_p8_rrtmgp 043 failed in check_result failed
[RT] Error: Test control_p8_rrtmgp 043 failed in run_test failed
[RT] Error: Test merra2_thompson 044 failed in check_result failed
[RT] Error: Test merra2_thompson 044 failed in run_test failed
[RT] Error: Test rrfs_v1beta 066 failed in check_result failed
[RT] Error: Test rrfs_v1beta 066 failed in run_test failed
[RT] Error: Test rrfs_v1nssl 067 failed in check_result failed
[RT] Error: Test rrfs_v1nssl 067 failed in run_test failed
[RT] Error: Test rrfs_v1nssl_nohailnoccn 068 failed in check_result failed
[RT] Error: Test rrfs_v1nssl_nohailnoccn 068 failed in run_test failed
[RT] Error: Test control_debug_p8 090 failed in check_result failed
[RT] Error: Test control_debug_p8 090 failed in run_test failed
[RT] Error: Test rrfs_v1beta_debug 105 failed in check_result failed
[RT] Error: Test rrfs_v1beta_debug 105 failed in run_test failed
[RT] Error: Test control_p8_atmlnd_sbs 152 failed in check_result failed
[RT] Error: Test control_p8_atmlnd_sbs 152 failed in run_test failed
[RT] Error: Test atmaero_control_p8 154 failed in check_result failed
[RT] Error: Test atmaero_control_p8 154 failed in run_test failed
[RT] Error: Test atmaero_control_p8_rad 155 failed in check_result failed
[RT] Error: Test atmaero_control_p8_rad 155 failed in run_test failed
[RT] Error: Test atmaero_control_p8_rad_micro 156 failed in check_result failed
[RT] Error: Test atmaero_control_p8_rad_micro 156 failed in run_test failed
[RT] Error: Test regional_atmaq 157 failed in check_result failed
[RT] Error: Test regional_atmaq 157 failed in run_test failed
[RT] Error: Test regional_atmaq_debug 158 failed in run_test failed
Please make changes and add the following label back: cheyenne-intel-RT

@dustinswales
Copy link
Copy Markdown
Member

Automated RT Failure Notification
Machine: cheyenne
Compiler: gnu
Job: BL
[BL] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1143720039/20221219071505/ufs-weather-model
[BL] Baseline creation and move successful
[RT] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1143720039/20221219074332/ufs-weather-model
Please make changes and add the following label back: cheyenne-gnu-BL

@dustinswales
Copy link
Copy Markdown
Member

Automated RT Failure Notification
Machine: cheyenne
Compiler: intel
Job: BL
[BL] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1143720039/20221219073010/ufs-weather-model
[BL] Error: Test cpld_control_c192_p8 003 failed in run_test failed
[BL] Error: Test rrfs_conus13km_hrrr_warm_debug 048 failed in run_test failed
[BL] Error: Test rrfs_conus13km_radar_tten_warm_debug 049 failed in run_test failed
[BL] Error: Test control_CubedSphereGrid_debug 051 failed in run_test failed
[BL] Error: Test control_stochy_debug 053 failed in run_test failed
[BL] Error: Test control_csawmg_debug 055 failed in run_test failed
[BL] Error: Test control_csawmgt_debug 056 failed in run_test failed
[BL] Error: Test control_ras_debug 057 failed in run_test failed
[BL] Error: Test control_diag_debug 058 failed in run_test failed
[BL] Error: Test control_debug_p8 059 failed in run_test failed
[BL] Error: Test regional_debug 060 failed in run_test failed
[BL] Error: Test rap_control_debug 061 failed in run_test failed
[BL] Error: Test hrrr_control_debug 062 failed in run_test failed
[BL] Error: Test rap_diag_debug 063 failed in run_test failed
[BL] Error: Test rap_cires_ugwp_debug 064 failed in run_test failed
[BL] Error: Test rap_lndp_debug 065 failed in run_test failed
[BL] Error: Test rap_flake_debug 066 failed in run_test failed
[BL] Error: Test rap_progcld_thompson_debug 067 failed in run_test failed
[BL] Error: Test rap_noah_debug 068 failed in run_test failed
[BL] Error: Test rap_sfcdiff_debug 070 failed in run_test failed
[BL] Error: Test rap_noah_sfcdiff_cires_ugwp_debug 071 failed in run_test failed
[BL] Error: Test hafs_regional_atm 081 failed in run_test failed
[BL] Error: Test hafs_regional_atm_thompson_gfdlsf 082 failed in run_test failed
[BL] Error: Test hafs_regional_atm_ocn 083 failed in run_test failed
[BL] Error: Test hafs_regional_atm_wav 084 failed in run_test failed
[BL] Error: Test hafs_regional_atm_ocn_wav 085 failed in run_test failed
[BL] Error: Test hafs_regional_1nest_atm 086 failed in run_test failed
[BL] Error: Test hafs_regional_telescopic_2nests_atm 087 failed in run_test failed
[BL] Error: Test hafs_global_multiple_4nests_atm 089 failed in run_test failed
[BL] Error: Test hafs_regional_specified_moving_1nest_atm 090 failed in run_test failed
[BL] Error: Test hafs_regional_storm_following_1nest_atm 091 failed in run_test failed
[BL] Error: Test hafs_regional_storm_following_1nest_atm_ocn 092 failed in run_test failed
[BL] Error: Test hafs_regional_storm_following_1nest_atm_ocn_wav 093 failed in run_test failed
[BL] Error: Test hafs_global_storm_following_1nest_atm 094 failed in run_test failed
[BL] Error: Test hafs_regional_docn 095 failed in run_test failed
[BL] Error: Test hafs_regional_docn_oisst 096 failed in run_test failed
[BL] Error: Test hafs_regional_datm_cdeps 097 failed in run_test failed
[BL] Error: Test datm_cdeps_mx025_cfsr 105 failed in run_test failed
[BL] Error: Test datm_cdeps_mx025_gefs 106 failed in run_test failed
[BL] Error: Test atmaero_control_p8 113 failed in run_test failed
[BL] Error: Test atmaero_control_p8_rad 114 failed in run_test failed
[BL] Error: Test atmaero_control_p8_rad_micro 115 failed in run_test failed
[BL] Error: Test regional_atmaq_debug 117 failed in run_test failed
[BL] Error: Test compile_022 failed in run_compile failed
[BL] Error: Test compile_023 failed in run_compile failed
[BL] Error: Test compile_025 failed in run_compile failed
Please make changes and add the following label back: cheyenne-intel-BL

@dustinswales
Copy link
Copy Markdown
Member

Automated RT Failure Notification
Machine: cheyenne
Compiler: gnu
Job: BL
[BL] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1143720039/20221219111507/ufs-weather-model
[BL] Baseline creation and move successful
[RT] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1143720039/20221219115425/ufs-weather-model
[RT] Error: Test control_debug 023 failed in run_test failed
[RT] Error: Test rap_diag_debug 028 failed in run_test failed
[RT] Error: Test rap_progcld_thompson_debug 030 failed in run_test failed
[RT] Error: Test rrfs_v1beta_debug 031 failed in run_test failed
[RT] Error: Test control_ras_debug 032 failed in run_test failed
[RT] Error: Test control_stochy_debug 033 failed in run_test failed
[RT] Error: Test control_debug_p8 034 failed in run_test failed
[RT] Error: Test rrfs_conus13km_hrrr_warm_debug 035 failed in run_test failed
[RT] Error: Test rap_control_dyn64_phy32 045 failed in run_test failed
[RT] Error: Test rap_control_debug_dyn32_phy32 046 failed in run_test failed
[RT] Error: Test hrrr_control_debug_dyn32_phy32 047 failed in run_test failed
[RT] Error: Test cpld_control_p8 049 failed in run_test failed
[RT] Error: Test cpld_control_nowave_noaero_p8 050 failed in run_test failed
[RT] Error: Test cpld_debug_p8 051 failed in run_test failed
Please make changes and add the following label back: cheyenne-gnu-BL

@grantfirl
Copy link
Copy Markdown
Collaborator Author

@dustinswales It looks like we're running into a full disk quota on Cheyenne. Could you help to clear some space?

@dustinswales
Copy link
Copy Markdown
Member

Automated RT Failure Notification
Machine: cheyenne
Compiler: intel
Job: BL
[BL] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1143720039/20221219113009/ufs-weather-model
[BL] Error: Test rrfs_conus13km_hrrr_warm_debug 048 failed in run_test failed
[BL] Error: Test control_csawmg_debug 055 failed in run_test failed
[BL] Error: Test control_csawmgt_debug 056 failed in run_test failed
[BL] Error: Test control_ras_debug 057 failed in run_test failed
[BL] Error: Test control_diag_debug 058 failed in run_test failed
[BL] Error: Test control_debug_p8 059 failed in run_test failed
[BL] Error: Test rap_control_debug 061 failed in run_test failed
[BL] Error: Test hrrr_control_debug 062 failed in run_test failed
[BL] Error: Test rap_diag_debug 063 failed in run_test failed
[BL] Error: Test rap_cires_ugwp_debug 064 failed in run_test failed
[BL] Error: Test rap_lndp_debug 065 failed in run_test failed
[BL] Error: Test rap_flake_debug 066 failed in run_test failed
[BL] Error: Test rap_progcld_thompson_debug 067 failed in run_test failed
[BL] Error: Test rap_noah_debug 068 failed in run_test failed
[BL] Error: Test rap_sfcdiff_debug 070 failed in run_test failed
[BL] Error: Test rap_noah_sfcdiff_cires_ugwp_debug 071 failed in run_test failed
[BL] Error: Test rrfs_v1beta_debug 072 failed in run_test failed
[BL] Error: Test hafs_regional_atm_ocn 083 failed in run_test failed
[BL] Error: Test hafs_regional_atm_wav 084 failed in run_test failed
[BL] Error: Test hafs_regional_atm_ocn_wav 085 failed in run_test failed
[BL] Error: Test hafs_global_multiple_4nests_atm 089 failed in run_test failed
[BL] Error: Test hafs_regional_storm_following_1nest_atm 091 failed in run_test failed
[BL] Error: Test hafs_regional_storm_following_1nest_atm_ocn 092 failed in run_test failed
[BL] Error: Test hafs_regional_storm_following_1nest_atm_ocn_wav 093 failed in run_test failed
[BL] Error: Test hafs_regional_docn 095 failed in run_test failed
[BL] Error: Test hafs_regional_docn_oisst 096 failed in run_test failed
[BL] Error: Test hafs_regional_datm_cdeps 097 failed in run_test failed
[BL] Error: Test control_p8_atmlnd_sbs 111 failed in run_test failed
[BL] Error: Test control_atmwav 112 failed in run_test failed
[BL] Error: Test atmaero_control_p8 113 failed in run_test failed
[BL] Error: Test atmaero_control_p8_rad 114 failed in run_test failed
[BL] Error: Test atmaero_control_p8_rad_micro 115 failed in run_test failed
[BL] Error: Test regional_atmaq 116 failed in run_test failed
[BL] Error: Test regional_atmaq_debug 117 failed in run_test failed
Please make changes and add the following label back: cheyenne-intel-BL

@dustinswales
Copy link
Copy Markdown
Member

Automated RT Failure Notification
Machine: cheyenne
Compiler: intel
Job: BL
[BL] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1143720039/20221219094507/ufs-weather-model
[BL] Baseline creation and move successful
[RT] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1143720039/20221219105140/ufs-weather-model
[RT] Error: Test cpld_control_p8 001 failed in check_result failed
[RT] Error: Test cpld_control_p8 001 failed in run_test failed
[RT] Error: Test cpld_2threads_p8 003 failed in check_result failed
[RT] Error: Test cpld_2threads_p8 003 failed in run_test failed
[RT] Error: Test cpld_esmfthreads_p8 004 failed in check_result failed
[RT] Error: Test cpld_esmfthreads_p8 004 failed in run_test failed
[RT] Error: Test cpld_decomp_p8 005 failed in check_result failed
[RT] Error: Test cpld_decomp_p8 005 failed in run_test failed
[RT] Error: Test cpld_mpi_p8 006 failed in check_result failed
[RT] Error: Test cpld_mpi_p8 006 failed in run_test failed
[RT] Error: Test cpld_control_ciceC_p8 007 failed in check_result failed
[RT] Error: Test cpld_control_ciceC_p8 007 failed in run_test failed
[RT] Error: Test cpld_control_c192_p8 008 failed in run_test failed
[RT] Error: Test cpld_control_noaero_p8 010 failed in run_test failed
[RT] Error: Test cpld_control_nowave_noaero_p8 011 failed in check_result failed
[RT] Error: Test cpld_control_nowave_noaero_p8 011 failed in run_test failed
[RT] Error: Test cpld_debug_p8 012 failed in check_result failed
[RT] Error: Test cpld_debug_p8 012 failed in run_test failed
[RT] Error: Test cpld_debug_noaero_p8 013 failed in check_result failed
[RT] Error: Test cpld_debug_noaero_p8 013 failed in run_test failed
[RT] Error: Test cpld_control_noaero_p8_agrid 014 failed in check_result failed
[RT] Error: Test cpld_control_noaero_p8_agrid 014 failed in run_test failed
[RT] Error: Test cpld_control_c48 015 failed in check_result failed
[RT] Error: Test cpld_control_c48 015 failed in run_test failed
[RT] Error: Test cpld_warmstart_c48 016 failed in run_test failed
[RT] Error: Test control 018 failed in check_result failed
[RT] Error: Test control 018 failed in run_test failed
[RT] Error: Test control_decomp 019 failed in check_result failed
[RT] Error: Test control_decomp 019 failed in run_test failed
[RT] Error: Test control_2dwrtdecomp 020 failed in check_result failed
[RT] Error: Test control_2dwrtdecomp 020 failed in run_test failed
[RT] Error: Test control_2threads 021 failed in check_result failed
[RT] Error: Test control_2threads 021 failed in run_test failed
[RT] Error: Test control_fhzero 023 failed in run_test failed
[RT] Error: Test control_CubedSphereGrid 024 failed in check_result failed
[RT] Error: Test control_CubedSphereGrid 024 failed in run_test failed
[RT] Error: Test control_CubedSphereGrid_parallel 025 failed in check_result failed
[RT] Error: Test control_CubedSphereGrid_parallel 025 failed in run_test failed
[RT] Error: Test control_latlon 026 failed in check_result failed
[RT] Error: Test control_latlon 026 failed in run_test failed
[RT] Error: Test control_wrtGauss_netcdf_parallel 027 failed in check_result failed
[RT] Error: Test control_wrtGauss_netcdf_parallel 027 failed in run_test failed
[RT] Error: Test control_c48 028 failed in check_result failed
[RT] Error: Test control_c48 028 failed in run_test failed
[RT] Error: Test control_c192 029 failed in check_result failed
[RT] Error: Test control_c192 029 failed in run_test failed
[RT] Error: Test control_c384 030 failed in check_result failed
[RT] Error: Test control_c384 030 failed in run_test failed
[RT] Error: Test control_c384gdas 031 failed in check_result failed
[RT] Error: Test control_c384gdas 031 failed in run_test failed
[RT] Error: Test control_c384_progsigma 032 failed in run_test failed
[RT] Error: Test control_stochy 033 failed in check_result failed
[RT] Error: Test control_stochy 033 failed in run_test failed
[RT] Error: Test control_lndp 035 failed in check_result failed
[RT] Error: Test control_lndp 035 failed in run_test failed
[RT] Error: Test control_iovr4 036 failed in run_test failed
[RT] Error: Test control_iovr5 037 failed in check_result failed
[RT] Error: Test control_iovr5 037 failed in run_test failed
[RT] Error: Test control_p8 038 failed in check_result failed
[RT] Error: Test control_p8 038 failed in run_test failed
[RT] Error: Test control_p8_lndp 039 failed in check_result failed
[RT] Error: Test control_p8_lndp 039 failed in run_test failed
[RT] Error: Test control_decomp_p8 041 failed in run_test failed
[RT] Error: Test control_2threads_p8 042 failed in check_result failed
[RT] Error: Test control_2threads_p8 042 failed in run_test failed
[RT] Error: Test control_p8_rrtmgp 043 failed in check_result failed
[RT] Error: Test control_p8_rrtmgp 043 failed in run_test failed
[RT] Error: Test merra2_thompson 044 failed in check_result failed
[RT] Error: Test merra2_thompson 044 failed in run_test failed
[RT] Error: Test regional_control 045 failed in check_result failed
[RT] Error: Test regional_control 045 failed in run_test failed
[RT] Error: Test regional_decomp 047 failed in check_result failed
[RT] Error: Test regional_decomp 047 failed in run_test failed
[RT] Error: Test regional_2threads 048 failed in run_test failed
[RT] Error: Test regional_noquilt 049 failed in check_result failed
[RT] Error: Test regional_noquilt 049 failed in run_test failed
[RT] Error: Test regional_netcdf_parallel 050 failed in check_result failed
[RT] Error: Test regional_netcdf_parallel 050 failed in run_test failed
[RT] Error: Test regional_2dwrtdecomp 051 failed in check_result failed
[RT] Error: Test regional_2dwrtdecomp 051 failed in run_test failed
[RT] Error: Test regional_wofs 052 failed in check_result failed
[RT] Error: Test regional_wofs 052 failed in run_test failed
[RT] Error: Test rap_control 053 failed in check_result failed
[RT] Error: Test rap_control 053 failed in run_test failed
[RT] Error: Test rap_rrtmgp 054 failed in run_test failed
[RT] Error: Test regional_spp_sppt_shum_skeb 055 failed in check_result failed
[RT] Error: Test regional_spp_sppt_shum_skeb 055 failed in run_test failed
[RT] Error: Test rap_decomp 056 failed in check_result failed
[RT] Error: Test rap_decomp 056 failed in run_test failed
[RT] Error: Test rap_2threads 057 failed in check_result failed
[RT] Error: Test rap_2threads 057 failed in run_test failed
[RT] Error: Test rap_sfcdiff 059 failed in check_result failed
[RT] Error: Test rap_sfcdiff 059 failed in run_test failed
[RT] Error: Test rap_sfcdiff_decomp 060 failed in check_result failed
[RT] Error: Test rap_sfcdiff_decomp 060 failed in run_test failed
[RT] Error: Test hrrr_control 062 failed in check_result failed
[RT] Error: Test hrrr_control 062 failed in run_test failed
[RT] Error: Test hrrr_control_decomp 063 failed in check_result failed
[RT] Error: Test hrrr_control_decomp 063 failed in run_test failed
[RT] Error: Test hrrr_control_2threads 064 failed in check_result failed
[RT] Error: Test hrrr_control_2threads 064 failed in run_test failed
[RT] Error: Test rrfs_v1beta 066 failed in check_result failed
[RT] Error: Test rrfs_v1beta 066 failed in run_test failed
[RT] Error: Test rrfs_v1nssl 067 failed in check_result failed
[RT] Error: Test rrfs_v1nssl 067 failed in run_test failed
[RT] Error: Test rrfs_v1nssl_nohailnoccn 068 failed in check_result failed
[RT] Error: Test rrfs_v1nssl_nohailnoccn 068 failed in run_test failed
[RT] Error: Test rrfs_conus13km_hrrr_warm 069 failed in check_result failed
[RT] Error: Test rrfs_conus13km_hrrr_warm 069 failed in run_test failed
[RT] Error: Test rrfs_smoke_conus13km_hrrr_warm 070 failed in check_result failed
[RT] Error: Test rrfs_smoke_conus13km_hrrr_warm 070 failed in run_test failed
[RT] Error: Test rrfs_conus13km_radar_tten_warm 071 failed in check_result failed
[RT] Error: Test rrfs_conus13km_radar_tten_warm 071 failed in run_test failed
[RT] Error: Test rrfs_conus13km_hrrr_warm_2threads 072 failed in check_result failed
[RT] Error: Test rrfs_conus13km_hrrr_warm_2threads 072 failed in run_test failed
[RT] Error: Test rrfs_conus13km_radar_tten_warm_2threads 073 failed in check_result failed
[RT] Error: Test rrfs_conus13km_radar_tten_warm_2threads 073 failed in run_test failed
[RT] Error: Test control_csawmg 074 failed in run_test failed
[RT] Error: Test control_csawmgt 075 failed in check_result failed
[RT] Error: Test control_csawmgt 075 failed in run_test failed
[RT] Error: Test control_ras 076 failed in check_result failed
[RT] Error: Test control_ras 076 failed in run_test failed
[RT] Error: Test control_wam 077 failed in run_test failed
[RT] Error: Test rrfs_conus13km_hrrr_warm_debug 078 failed in check_result failed
[RT] Error: Test rrfs_conus13km_hrrr_warm_debug 078 failed in run_test failed
[RT] Error: Test rrfs_conus13km_radar_tten_warm_debug 079 failed in run_test failed
[RT] Error: Test control_debug 080 failed in check_result failed
[RT] Error: Test control_debug 080 failed in run_test failed
[RT] Error: Test control_2threads_debug 081 failed in check_result failed
[RT] Error: Test control_2threads_debug 081 failed in run_test failed
[RT] Error: Test control_CubedSphereGrid_debug 082 failed in check_result failed
[RT] Error: Test control_CubedSphereGrid_debug 082 failed in run_test failed
[RT] Error: Test control_wrtGauss_netcdf_parallel_debug 083 failed in check_result failed
[RT] Error: Test control_wrtGauss_netcdf_parallel_debug 083 failed in run_test failed
[RT] Error: Test control_stochy_debug 084 failed in check_result failed
[RT] Error: Test control_stochy_debug 084 failed in run_test failed
[RT] Error: Test control_lndp_debug 085 failed in run_test failed
[RT] Error: Test control_csawmg_debug 086 failed in check_result failed
[RT] Error: Test control_csawmg_debug 086 failed in run_test failed
[RT] Error: Test control_csawmgt_debug 087 failed in check_result failed
[RT] Error: Test control_csawmgt_debug 087 failed in run_test failed
[RT] Error: Test control_ras_debug 088 failed in check_result failed
[RT] Error: Test control_ras_debug 088 failed in run_test failed
[RT] Error: Test control_diag_debug 089 failed in check_result failed
[RT] Error: Test control_diag_debug 089 failed in run_test failed
[RT] Error: Test control_debug_p8 090 failed in run_test failed
[RT] Error: Test regional_debug 091 failed in check_result failed
[RT] Error: Test regional_debug 091 failed in run_test failed
[RT] Error: Test rap_control_debug 092 failed in check_result failed
[RT] Error: Test rap_control_debug 092 failed in run_test failed
[RT] Error: Test hrrr_control_debug 093 failed in check_result failed
[RT] Error: Test hrrr_control_debug 093 failed in run_test failed
[RT] Error: Test rap_unified_drag_suite_debug 094 failed in check_result failed
[RT] Error: Test rap_unified_drag_suite_debug 094 failed in run_test failed
[RT] Error: Test rap_diag_debug 095 failed in check_result failed
[RT] Error: Test rap_diag_debug 095 failed in run_test failed
[RT] Error: Test rap_cires_ugwp_debug 096 failed in check_result failed
[RT] Error: Test rap_cires_ugwp_debug 096 failed in run_test failed
[RT] Error: Test rap_unified_ugwp_debug 097 failed in check_result failed
[RT] Error: Test rap_unified_ugwp_debug 097 failed in run_test failed
[RT] Error: Test rap_lndp_debug 098 failed in check_result failed
[RT] Error: Test rap_lndp_debug 098 failed in run_test failed
[RT] Error: Test rap_flake_debug 099 failed in check_result failed
[RT] Error: Test rap_flake_debug 099 failed in run_test failed
[RT] Error: Test rap_progcld_thompson_debug 100 failed in check_result failed
[RT] Error: Test rap_progcld_thompson_debug 100 failed in run_test failed
[RT] Error: Test rap_noah_debug 101 failed in check_result failed
[RT] Error: Test rap_noah_debug 101 failed in run_test failed
[RT] Error: Test rap_rrtmgp_debug 102 failed in run_test failed
[RT] Error: Test rap_sfcdiff_debug 103 failed in run_test failed
[RT] Error: Test rap_noah_sfcdiff_cires_ugwp_debug 104 failed in run_test failed
[RT] Error: Test rrfs_v1beta_debug 105 failed in run_test failed
[RT] Error: Test control_wam_debug 106 failed in run_test failed
[RT] Error: Test regional_spp_sppt_shum_skeb_dyn32_phy32 107 failed in run_test failed
[RT] Error: Test rap_control_dyn32_phy32 108 failed in run_test failed
[RT] Error: Test hrrr_control_dyn32_phy32 109 failed in check_result failed
[RT] Error: Test hrrr_control_dyn32_phy32 109 failed in run_test failed
[RT] Error: Test rap_2threads_dyn32_phy32 110 failed in run_test failed
[RT] Error: Test hrrr_control_2threads_dyn32_phy32 111 failed in run_test failed
[RT] Error: Test hrrr_control_decomp_dyn32_phy32 112 failed in check_result failed
[RT] Error: Test hrrr_control_decomp_dyn32_phy32 112 failed in run_test failed
[RT] Error: Test rap_control_dyn64_phy32 115 failed in run_test failed
[RT] Error: Test rap_control_debug_dyn32_phy32 116 failed in run_test failed
[RT] Error: Test hrrr_control_debug_dyn32_phy32 117 failed in check_result failed
[RT] Error: Test hrrr_control_debug_dyn32_phy32 117 failed in run_test failed
[RT] Error: Test rap_control_dyn64_phy32_debug 118 failed in run_test failed
[RT] Error: Test hafs_regional_atm 119 failed in run_test failed
[RT] Error: Test hafs_regional_atm_thompson_gfdlsf 120 failed in run_test failed
[RT] Error: Test hafs_regional_atm_ocn 121 failed in run_test failed
[RT] Error: Test hafs_regional_atm_wav 122 failed in run_test failed
[RT] Error: Test hafs_regional_atm_ocn_wav 123 failed in run_test failed
[RT] Error: Test hafs_regional_1nest_atm 124 failed in run_test failed
[RT] Error: Test hafs_regional_telescopic_2nests_atm 125 failed in run_test failed
[RT] Error: Test hafs_global_1nest_atm 126 failed in run_test failed
[RT] Error: Test hafs_global_multiple_4nests_atm 127 failed in run_test failed
[RT] Error: Test hafs_regional_specified_moving_1nest_atm 128 failed in run_test failed
[RT] Error: Test hafs_regional_storm_following_1nest_atm 129 failed in run_test failed
[RT] Error: Test hafs_regional_storm_following_1nest_atm_ocn 130 failed in run_test failed
[RT] Error: Test hafs_regional_storm_following_1nest_atm_ocn_wav 131 failed in run_test failed
[RT] Error: Test hafs_global_storm_following_1nest_atm 132 failed in run_test failed
[RT] Error: Test hafs_regional_docn 133 failed in run_test failed
[RT] Error: Test hafs_regional_docn_oisst 134 failed in run_test failed
[RT] Error: Test hafs_regional_datm_cdeps 135 failed in run_test failed
[RT] Error: Test datm_cdeps_control_cfsr 136 failed in run_test failed
[RT] Error: Test datm_cdeps_control_gefs 138 failed in run_test failed
[RT] Error: Test datm_cdeps_iau_gefs 139 failed in run_test failed
[RT] Error: Test datm_cdeps_stochy_gefs 140 failed in run_test failed
[RT] Error: Test datm_cdeps_ciceC_cfsr 141 failed in run_test failed
[RT] Error: Test datm_cdeps_bulk_cfsr 142 failed in run_test failed
[RT] Error: Test datm_cdeps_bulk_gefs 143 failed in run_test failed
[RT] Error: Test datm_cdeps_mx025_cfsr 144 failed in run_test failed
[RT] Error: Test datm_cdeps_mx025_gefs 145 failed in run_test failed
[RT] Error: Test datm_cdeps_multiple_files_cfsr 146 failed in run_test failed
[RT] Error: Test datm_cdeps_3072x1536_cfsr 147 failed in run_test failed
[RT] Error: Test datm_cdeps_gfs 148 failed in run_test failed
[RT] Error: Test datm_cdeps_debug_cfsr 149 failed in run_test failed
[RT] Error: Test datm_cdeps_lnd_gswp3 150 failed in run_test failed
[RT] Error: Test control_atmwav 153 failed in run_test failed
[RT] Error: Test atmaero_control_p8 154 failed in run_test failed
[RT] Error: Test atmaero_control_p8_rad 155 failed in run_test failed
[RT] Error: Test atmaero_control_p8_rad_micro 156 failed in run_test failed
[RT] Error: Test regional_atmaq 157 failed in run_test failed
[RT] Error: Test regional_atmaq_debug 158 failed in run_test failed
[RT] Error: Test compile_022 failed in run_compile failed
Please make changes and add the following label back: cheyenne-intel-BL

@dustinswales
Copy link
Copy Markdown
Member

Automated RT Failure Notification
Machine: cheyenne
Compiler: gnu
Job: BL
[BL] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1143720039/20221219170008/ufs-weather-model
[BL] ERROR: Baseline location exists before creation:
/glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/NCAR/main-20221219/GNU
Please make changes and add the following label back: cheyenne-gnu-BL

@dustinswales
Copy link
Copy Markdown
Member

Automated RT Failure Notification
Machine: cheyenne
Compiler: intel
Job: BL
[BL] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1143720039/20221219170136/ufs-weather-model
[BL] Baseline creation and move successful
[RT] Repo location: /glade/scratch/epicufsrt/GMTB/ufs-weather-model/RT/auto_RT/Pull_Requests/1143720039/20221219182935/ufs-weather-model
[RT] Error: Test regional_atmaq_debug 158 failed in run_test failed
Please make changes and add the following label back: cheyenne-intel-BL

@dustinswales
Copy link
Copy Markdown
Member

@grantfirl regional_atmaq_debug Intel failed due to exceeding the WLCLK. Can you add more time to that job and kick off the BL tests again?

@grantfirl
Copy link
Copy Markdown
Collaborator Author

@dustinswales When autoRT posts a ERROR: Baseline location exists before creation message, do you actually have to log in and delete the existing baseline, or does the script automatically do this?

@dustinswales
Copy link
Copy Markdown
Member

@dustinswales When autoRT posts a ERROR: Baseline location exists before creation message, do you actually have to log in and delete the existing baseline, or does the script automatically do this?

@grantfirl Yeah. And I have to start the process over.
This is kind of fragile in that we could add the BL tag, which creates new baselines and runs the tests again to compare to the baselines, but a wallclock limit could be hit during that second part. This a) leaves an existing directory that needs to be manually removed before we can run the tests again (or we get the error you mentioned) and b) we also need to extend the WLCLK for the test that needs more time.

@grantfirl
Copy link
Copy Markdown
Collaborator Author

@ok, I didn't realize that the old baseline had to manually removed, and I haven't done that yet. I wonder if I do it now whether it will error out?

@grantfirl
Copy link
Copy Markdown
Collaborator Author

@dustinswales Since the baselines are being created by your user and the files have permissions of 644, I can't delete them. If I want to have that ability, how should we accomplish this? Put something in the scripts to chmod all the files in there to 664?

This was referenced Dec 20, 2022
on-behalf-of @NCAR <dswales@ucar.edu>
@grantfirl
Copy link
Copy Markdown
Collaborator Author

@dustinswales I think that this should be ready to merge as soon as you approve this and submodule PRs, right?

@dustinswales
Copy link
Copy Markdown
Member

@grantfirl Good to go. (I will add a chmod step in the auto-RT step so you can do cleanup if necessary)

@BrianCurtis-NOAA
Copy link
Copy Markdown

@ok, I didn't realize that the old baseline had to manually removed, and I haven't done that yet. I wonder if I do it now whether it will error out?

The original reasoning for this functionality was in case the autoRT script got activated on accident it wouldn't overwrite current baselines. So we just set it to see if the directory exists, and if it did, it automatically failed.

@grantfirl grantfirl merged commit 1379e69 into NCAR:main Dec 20, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants