Testing of the Marine DA jjobs

[guillaumevernieres]

Still a bit of a work todo but good enough for a draft PR.

Addition of 3 ctests related to running with the g-w

• Copy/paste of @CoryMartin-NOAA 's setup_expt.py test adjusted to soca
• jjob for the Marine DA prep step (create yaml's, stage obs and bkg's, ...)
• jjob for the Marine DA run step (gridgen, static B init., run var, create iau increment, ...)

Stuff left to do

• Only tested in a singularity container
  - As I type this I'm wondering if the changes I made to the g-w are needed (probably not)
• The prep jjob fails when run twice in a row, simple fix but still in the todo list
• ...

Issues addressed

• fixes 5deg Marine 3DVar ctest using j-job #232

[CoryMartin-NOAA]

a few comments/questions

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Wed Dec  7 19:47:33 UTC 2022 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Wed Dec  7 20:43:53 UTC 2022
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Wed Dec  7 20:45:39 UTC 2022
Tests: 89% tests passed, 3 tests failed out of 28
	1291 - test_gdasapp_soca_ana_run (Failed)
	1293 - test_gdasapp_soca_jjob_soca_prep (Failed)
	1294 - test_gdasapp_soca_jjob_soca_run (Failed)
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/234/global-workflow/sorc/gdas.cd/build/log.ctest

[guillaumevernieres]

Looks like I hard-coded CONTAINER in config.base, so the new tests are failing and ... the file name is 136 character long, so it's truncated when going through fms io. I'm not sure why that bug didn't show up the last time we ran the ci.

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Wed Dec  7 23:15:53 UTC 2022 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Wed Dec  7 23:57:29 UTC 2022
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Wed Dec  7 23:59:48 UTC 2022
Tests: 86% tests passed, 4 tests failed out of 28
	1290 - test_gdasapp_setup_cycled_exp (Failed)
	1301 - test_gdasapp_soca_ana_run (Failed)
	1303 - test_gdasapp_soca_jjob_soca_prep (Failed)
	1304 - test_gdasapp_soca_jjob_soca_run (Failed)
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/234/global-workflow/sorc/gdas.cd/build/log.ctest

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Thu Dec  8 21:30:38 UTC 2022 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Thu Dec  8 21:54:02 UTC 2022
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Thu Dec  8 21:55:44 UTC 2022
Tests: 90% tests passed, 3 tests failed out of 29
	1290 - test_gdasapp_setup_cycled_exp (Failed)
	1304 - test_gdasapp_soca_jjob_soca_prep (Failed)
	1305 - test_gdasapp_soca_jjob_soca_run (Failed)
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/234/global-workflow/sorc/gdas.cd/build/log.ctest

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Wed Dec 14 03:00:36 UTC 2022 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Wed Dec 14 03:24:51 UTC 2022
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Wed Dec 14 03:28:05 UTC 2022
Tests: 97% tests passed, 1 tests failed out of 31
	1298 - test_gdasapp_setup_cycled_exp (Failed)
Tests: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/234/global-workflow/sorc/gdas.cd/build/log.ctest

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Tue Dec 13 21:00:54 CST 2022 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Tue Dec 13 21:28:23 CST 2022
---------------------------------------------------
Tests:                                  *Failed*
Tests: Failed at Tue Dec 13 21:34:32 CST 2022
Tests: 97% tests passed, 1 tests failed out of 31
	1298 - test_gdasapp_setup_cycled_exp (Failed)
Tests: see output at /work2/noaa/stmp/cmartin/CI/GDASApp/workflow/PR/234/global-workflow/sorc/gdas.cd/build/log.ctest

[guillaumevernieres]

We shall see, but I think all our test should be re-runable (it's a word ... ) now.

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Wed Dec 14 12:30:53 UTC 2022 on hfe07
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Wed Dec 14 12:55:08 UTC 2022
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Wed Dec 14 12:58:49 UTC 2022
Tests: 100% tests passed, 0 tests failed out of 31

[emcbot]

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Wed Dec 14 06:30:49 CST 2022 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Wed Dec 14 06:59:12 CST 2022
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Wed Dec 14 07:04:49 CST 2022
Tests: 100% tests passed, 0 tests failed out of 31

[guillaumevernieres]

Re-requesting your review @CoryMartin-NOAA , I had to add a few changes to make some of the ctest "re-runable".