Add capability to run diff resolutions for marine anl and background by guillaumevernieres · Pull Request #3238 · NOAA-EMC/global-workflow

guillaumevernieres · 2025-01-16T18:39:31Z

Description

This PR mostly includes configuration changes to allow the use a B-matrix at a defined resolution that may or may not be the same as the background.

Changes of note:

Addition of an interpolation step in the B-matrix job
version update for the fix files
the gdas.cd update should be the correct one now.

Resolves Implement the marine analysis at a resolution different from the deterministic #3230
Resolves Background error at 1/2 deg GDASApp#1375

Dependencies

Type of change

Bug fix (fixes something broken)
New feature (adds functionality)
Maintenance (code refactor, clean-up, new CI test, etc.)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO
Does this change require a documentation update? NO
Does this change require an update to any of the following submodules? YES (If YES, please add a link to any PRs that are pending.)
- EMC verif-global
- GDAS
- GFS-utils
- GSI
- GSI-monitor
- GSI-utils
- UFS-utils
- UFS-weather-model
- wxflow

How has this been tested?

Subset of the ci tests on hercules and hera

Checklist

…soca2cice

Updates to cp4

DavidNew-NOAA

Looks good except for the one comment

TerrenceMcGuinness-NOAA · 2025-02-11T16:14:15Z

Hera when into maintenance just the the experiments where being created and all the jobs failed. Removing the Running Label

RussTreadon-NOAA · 2025-02-12T13:31:49Z

Do we have an estimate as to when this PR will be merged into g-w develop?

Attention: @WalterKolczynski-NOAA , @aerorahul , @DavidNew-NOAA

WalterKolczynski-NOAA · 2025-02-13T14:30:00Z

@RussTreadon-NOAA We're waiting for some additional dependency fixes for the cleanup job to come in before running CI. The previous PR didn't get them all.

RussTreadon-NOAA · 2025-02-13T14:37:28Z

Thank you @WalterKolczynski-NOAA for the update. Two questions

Any idea when the additional dependency fixes for the cleanup job will be ready?
Do the additional dependency fixes for the cleanup job invalidate the g-w CI that was run over the weekend?

This PR is a blocker for subsequent marine DA changes for GFS v17 (e.g., GDASApp PR #1494).

WalterKolczynski-NOAA · 2025-02-13T15:50:43Z

Thank you @WalterKolczynski-NOAA for the update. Two questions

Any idea when the additional dependency fixes for the cleanup job will be ready?

Do the additional dependency fixes for the cleanup job invalidate the g-w CI that was run over the weekend?

This PR is a blocker for subsequent marine DA changes for GFS v17 (e.g., GDASApp PR #1494).

There's already a draft PR. Getting it fixed is a top priority.

RussTreadon-NOAA · 2025-02-13T15:57:33Z

Got it @WalterKolczynski-NOAA .

I assume the draft PR to which you refer is PR #3319. Still not clear to my why this PR, #3238, can not move forward until #3319 is merged into develop. Do we need to rerun g-w CI for this PR after PR #3319 is merged into develop?

WalterKolczynski-NOAA · 2025-02-13T16:38:04Z

Got it @WalterKolczynski-NOAA .

I assume the draft PR to which you refer is PR #3319. Still not clear to my why this PR, #3238, can not move forward until #3319 is merged into develop. Do we need to rerun g-w CI for this PR after PR #3319 is merged into develop?

CI has been failing in other PRs because the cleanup job runs prematurely, deleting working directories while jobs are still running. Every attempt at (automated) CI on this PR has also failed (before the dependency issues). I want at least one automated test to succeed here before merging.

RussTreadon-NOAA · 2025-02-14T22:55:08Z

@WalterKolczynski-NOAA , @aerorahul : Is there anything I can do to help this PR move forward? GDASApp CI fails when we run marine DA tests because those tests assume this PR is in g-w develop.

RussTreadon-NOAA · 2025-02-19T11:45:05Z

@WalterKolczynski-NOAA , @aerorahul : What can I do to move this PR forward? Is this PR waiting for other PRs to be merged into develop? GDASApp CI fails when we run marine DA tests because GDASApp tests assume this PR is in g-w develop.

shlyaeva · 2025-02-19T13:25:33Z

I'm happy to assist in any way to move this forward too (we also have several developments waiting for this to get merged); please let me know if there's anything I can do.

RussTreadon-NOAA · 2025-02-19T13:27:42Z

Thank you @shlyaeva for (a) offering to help and (b) noting that PR has become a blocker for ongoing development.

aerorahul · 2025-02-19T13:31:43Z

I think @WalterKolczynski-NOAA was waiting for the PR #3319 to me merged. See this. PR #3319 was merged yesterday and this PR was updated with that development yesterday. I think the CI can be launched on this PR to move this along.

RussTreadon-NOAA · 2025-02-19T13:35:26Z

Thank you @aerorahul for launching g-w CI on Hera and Hercules. Will success on these machines be sufficient or do we also need to launch g-w CI on WCOSS2 (Cactus)?

aerorahul · 2025-02-19T13:40:51Z

Hera and Hercules are enough; especially the automatic CI on WCOSS2 does not work for WCDA test (that was my understanding; happy to be corrected)

aerorahul · 2025-02-19T13:41:58Z

The WCDA test is also set to be skipped on WCOSS2.

DavidHuber-NOAA · 2025-02-19T18:35:06Z

Manual CI has started on Hera. The clone is located in /scratch1/NCEPDEV/global/David.Huber/GW/pr_3238 and tests will be run in /scratch1/NCEPDEV/global/David.Huber/pr/3238.

emcbot · 2025-02-19T19:00:45Z

CI Passed on Hercules in Build# 1
Built and ran in directory /work2/noaa/global/CI/HERCULES/3238


Experiment C48_ATM_b3782c9d Completed 1 Cycles: *SUCCESS* at Wed Feb 19 09:22:01 CST 2025
Experiment C48mx500_hybAOWCDA_b3782c9d Completed 2 Cycles: *SUCCESS* at Wed Feb 19 09:52:22 CST 2025
Experiment C96_S2SWA_gefs_replay_ics_b3782c9d Completed 1 Cycles: *SUCCESS* at Wed Feb 19 10:04:49 CST 2025
Experiment C96C48_hybatmDA_b3782c9d Completed 3 Cycles: *SUCCESS* at Wed Feb 19 11:11:19 CST 2025
Experiment C96_atm3DVar_b3782c9d Completed 3 Cycles: *SUCCESS* at Wed Feb 19 11:11:21 CST 2025
Experiment C48mx500_3DVarAOWCDA_b3782c9d Completed 2 Cycles: *SUCCESS* at Wed Feb 19 11:48:12 CST 2025
Experiment C48_S2SWA_gefs_b3782c9d Completed 1 Cycles: *SUCCESS* at Wed Feb 19 12:43:07 CST 2025
Experiment C48_S2SW_b3782c9d Completed 1 Cycles: *SUCCESS* at Wed Feb 19 13:00:32 CST 2025

RussTreadon-NOAA · 2025-02-19T19:51:35Z

WCOSS2 g-w CI

Since GFS v17 includes JEDI components install guillaumevernieres:feature/lowres-b at b3782c9 on Cactus. Set up and run g-w CI with the following results

/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_ATM_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Feb 19 2025 14:39:33    Feb 19 2025 15:45:18
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_3DVarAOWCDA_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Feb 19 2025 14:39:35    Feb 19 2025 14:55:29
202103250000        Done    Feb 19 2025 14:39:35    Feb 19 2025 16:40:27
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48mx500_hybAOWCDA_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103241800        Done    Feb 19 2025 14:39:38    Feb 19 2025 14:55:36
202103250000        Done    Feb 19 2025 14:39:38    Feb 19 2025 16:05:34
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SWA_gefs_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Feb 19 2025 14:39:41    Feb 19 2025 16:35:52
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C48_S2SW_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202103231200        Done    Feb 19 2025 14:39:44    Feb 19 2025 15:50:35
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_atm3DVar_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Feb 19 2025 14:39:46    Feb 19 2025 15:01:05
202112210000        Done    Feb 19 2025 14:39:46    Feb 19 2025 17:10:40
202112210600        Done    Feb 19 2025 14:39:46    Feb 19 2025 16:50:38
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmaerosnowDA_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201200        Done    Feb 19 2025 14:39:48    Feb 19 2025 15:01:08
202112201800        Done    Feb 19 2025 14:39:48    Feb 19 2025 17:05:36
202112210000        Done    Feb 19 2025 14:39:48    Feb 19 2025 16:50:42
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_hybatmDA_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202112201800        Done    Feb 19 2025 14:39:50    Feb 19 2025 15:01:12
202112210000        Done    Feb 19 2025 14:39:50    Feb 19 2025 16:45:45
202112210600        Done    Feb 19 2025 14:39:50    Feb 19 2025 16:50:46
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96C48_ufs_hybatmDA_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202402231800        Done    Feb 19 2025 14:39:52    Feb 19 2025 15:01:16
202402240000        Done    Feb 19 2025 14:39:52    Feb 19 2025 19:40:25
202402240600        Done    Feb 19 2025 14:39:52    Feb 19 2025 19:45:06
 
/lfs/h2/emc/ptmp/russ.treadon/EXPDIR/C96_S2SWA_gefs_replay_ics_pr3238
   CYCLE         STATE           ACTIVATED              DEACTIVATED     
202011010000        Done    Feb 19 2025 14:39:56    Feb 19 2025 15:41:03

All jobs in all cases successfully ran to completion.

Note that WCOSS2 g-w CI on Cactus was set up to run the same jobs that are run on Hera. Thus, hybatmaerosnowDA, 3DVarAOWCDA, hybAOWCDA, and ufs_hybatmDA were run on Cactus. These cases exercise various JEDI (GDASApp) capabilities.

RussTreadon-NOAA · 2025-02-19T19:59:37Z

@WalterKolczynski-NOAA, @DavidHuber-NOAA , and @aerorahul : g-w CI successfully ran on Cactus. The CI-Wcoss2_Passed label can be applied to this PR.

RussTreadon-NOAA · 2025-02-20T11:47:44Z

@DavidHuber-NOAA and @TerrenceMcGuinness-NOAA : While the CI-Hera-Running label remains, squeue on Hera does not return any pending or running jobs associated with this PR. Did Hera CI testing complete?

DavidHuber-NOAA · 2025-02-20T12:22:30Z

@RussTreadon-NOAA All tests passed except C48_ATM. This was due to an operator error (me). I am rerunning this test now.

DavidHuber-NOAA · 2025-02-20T13:10:43Z

The C48_ATM test case and all other tests have now completed. Setting CI-Hera-Passed label.

RussTreadon-NOAA · 2025-02-20T13:12:51Z

Thank you @DavidHuber-NOAA for guiding this PR through Hera g-w CI.

guillaumevernieres and others added 13 commits December 18, 2024 10:54

resources, no gfs, gdas dev

edbd0c9

Switch to using updating jedi hashes + soca and jcb-gdas changes for …

b565bef

…soca2cice

Bugfix some resources

e1a4fd4

change paths to hera + add dmpdir

e284329

fix the DMPDIR specification

7c37d0c

Merge pull request #9 from shlyaeva/feature/cp4

5990348

Updates to cp4

...

e95d993

Merge branch 'develop' into feature/cp4

8aa849f

low-res bmat works but still wip

fd54e37

Merge branch 'develop' into feature/cp4

4184e87

B geom in analysis task

7c2c058

use gdas.x for chg res

e62e9d0

norms

0699116

guillaumevernieres mentioned this pull request Jan 16, 2025

Add code changes for the implementation of the low-resolution B NOAA-EMC/GDASApp#1441

Merged

10 tasks

testing hack, revert when new fix files are staged

d736972

guillaumevernieres requested review from DavidNew-NOAA and shlyaeva January 16, 2025 21:01

fixed wrong exec

0c5ed87

DavidNew-NOAA requested changes Jan 17, 2025

View reviewed changes

Comment thread sorc/link_workflow.sh Outdated

guillaumevernieres added 4 commits January 21, 2025 10:10

bugfix of low res geom

29e0c78

fixed pynorms

3303119

removed hack

534ab08

Merge branch 'develop' into feature/lowres-b

c62f43a

guillaumevernieres added the JEDI Feature development to support JEDI-based DA label Jan 23, 2025

guillaumevernieres requested a review from DavidNew-NOAA January 23, 2025 18:27

DavidNew-NOAA previously approved these changes Jan 23, 2025

View reviewed changes

guillaumevernieres added 3 commits January 27, 2025 12:47

Merge branch 'develop' into feature/lowres-b

11b4f12

...

c90ba0c

...

9a693bf

guillaumevernieres dismissed DavidNew-NOAA’s stale review via 9a693bf January 27, 2025 19:10

guillaumevernieres mentioned this pull request Feb 12, 2025

Add a depth dependent e-fold scale NOAA-EMC/GDASApp#1494

Merged

6 tasks

Merge branch 'develop' into feature/lowres-b

c33da06

RussTreadon-NOAA mentioned this pull request Feb 14, 2025

Add options to inflate the ensemble while recentering NOAA-EMC/GDASApp#1501

Merged

6 tasks

Merge branch 'develop' into feature/lowres-b

4f35faf

Merge branch 'develop' into feature/lowres-b

b3782c9

DavidHuber-NOAA mentioned this pull request Feb 20, 2025

Update high resolution tests and fix minor wave issues #3289

Merged

13 tasks

RussTreadon-NOAA mentioned this pull request Feb 23, 2025

gdas_aeroanlgenb failure in 20250222 stable-nightly testing NOAA-EMC/GDASApp#1510

Closed

Conversation

guillaumevernieres commented Jan 16, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Changes of note:

Dependencies

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

DavidNew-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

TerrenceMcGuinness-NOAA commented Feb 11, 2025

Uh oh!

RussTreadon-NOAA commented Feb 12, 2025

Uh oh!

WalterKolczynski-NOAA commented Feb 13, 2025

Uh oh!

RussTreadon-NOAA commented Feb 13, 2025

Uh oh!

WalterKolczynski-NOAA commented Feb 13, 2025

Uh oh!

RussTreadon-NOAA commented Feb 13, 2025

Uh oh!

WalterKolczynski-NOAA commented Feb 13, 2025

Uh oh!

RussTreadon-NOAA commented Feb 14, 2025

Uh oh!

RussTreadon-NOAA commented Feb 19, 2025

Uh oh!

shlyaeva commented Feb 19, 2025

Uh oh!

RussTreadon-NOAA commented Feb 19, 2025

Uh oh!

aerorahul commented Feb 19, 2025

Uh oh!

RussTreadon-NOAA commented Feb 19, 2025

Uh oh!

aerorahul commented Feb 19, 2025

Uh oh!

aerorahul commented Feb 19, 2025

Uh oh!

DavidHuber-NOAA commented Feb 19, 2025

Uh oh!

emcbot commented Feb 19, 2025

Uh oh!

RussTreadon-NOAA commented Feb 19, 2025

Uh oh!

RussTreadon-NOAA commented Feb 19, 2025

Uh oh!

RussTreadon-NOAA commented Feb 20, 2025

Uh oh!

DavidHuber-NOAA commented Feb 20, 2025

Uh oh!

DavidHuber-NOAA commented Feb 20, 2025

Uh oh!

RussTreadon-NOAA commented Feb 20, 2025

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

11 participants

guillaumevernieres commented Jan 16, 2025 •

edited

Loading