Prepare observations for the updates to the ensemble members. by jiaruidong2017 · Pull Request #2416 · NOAA-EMC/global-workflow

jiaruidong2017 · 2024-03-20T13:31:25Z

Description

This PR adds in initial configurations files and jobs to prepare the snow observations for the JEDI based snow DA updates to the EnkF ensemble members. Because IMS observations are background dependent for each ensemble member, we plan to put snow observations in the ensemble directory for the later snow analysis.

Changes to the config.resources for the new task (add epsn)
Changes to the workflow tasks for the new task (add epsn)
Addition of new config file for snow DA related options (add config.epsn)

Because the snow observations are fetched from the COM_OBS in the gdas deterministic analysis and are saved into the COM_OBS in the enkfgdas analysis for the enkfgdas run, we have to define two different COM_OBS directories (COMIN_OBS vs COMOUT_OBS) and two obs basename heads (OPREFIX_IN vs OPREFIX_OUT) to identify the enkf analysis from the gdas deterministic analysis. The MEMDIR is added to COM_OBS to identify among the different enkf members. Other minor changes include to add CASE in the IMS filename.

This PR depends on NOAA-EMC/GDASApp#998.

Type of change

New feature (adds jobs to prepare the snow obs for the enkf analysis)

Change characteristics

Is this a breaking change (a change in existing functionality)? NO

How has this been tested?

Hybrid snow DA test on Hera was successful

Checklist

Any dependent changes have been merged and published
My code follows the style guidelines of this project
I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
My changes generate no new warnings
New and existing tests pass with my changes
I have made corresponding changes to the documentation if necessary

WalterKolczynski-NOAA

Conditionally approved pending successful completion of CI tests.

WalterKolczynski-NOAA · 2024-03-27T03:59:39Z

We'll need to wait for #2421 to be merged to test, since the relevant tests do not currently run on Orion and the CI system is set up to run on Rocky on Hera.

jiaruidong2017 · 2024-03-27T14:15:38Z

We'll need to wait for #2421 to be merged to test, since the relevant tests do not currently run on Orion and the CI system is set up to run on Rocky on Hera.

@WalterKolczynski-NOAA Please hold the CI test until the merge of NOAA-EMC/GDASApp#998 and the updates of the commit hash from GDASApp. Thanks.

CoryMartin-NOAA · 2024-04-01T18:52:09Z

 ##############################################
 # Generate COM variables from templates
-YMD=${PDY} HH=${cyc} generate_com -rx COM_OBS
+RUN=${oCDUMP} MEMDIR="" YMD=${PDY} HH=${cyc} generate_com -rx COM_OBS_IN:COM_OBS_TMPL


@aerorahul @WalterKolczynski-NOAA are you both okay with these additional variables? Something about this I don't particularly like but I don't have a good suggestion right now either.

There are a couple things I don't like here, the main one being this will not be acceptable to operations where COM_OBS_TMPL points to the obsproc space and should be strictly input for GFS. The out needs to use a different template (probably a new COM_SNOW_PREP_TMPL).

Also, if IN and OUT are going to be used in the variable name, would prefer to have it be COMIN_* and COMOUT_*. That is the pattern we plan on using when we add IN and OUT to the rest of the workflow.

Thanks for the good suggestions, and I have made changes from COM_OBS_IN/OUT to COMIN_OBS and COMOUT_OBS. I have no idea and wait for the suggestions on using a new COM_SNOW_PREP_TMPL or not.

CoryMartin-NOAA · 2024-04-03T19:12:14Z

+  source "${HOMEgfs}/ush/jjob_header.sh" -e "prepsnowobs" -c "base prepsnowobs"
+fi
+
+# Restart conditions for GFS cycle come from GDAS


what about enkfgfs?

I have no idea about enkfgfs.

CoryMartin-NOAA · 2024-04-03T19:13:15Z

+[[ ${RUN} == "gfs" ]] && export rCDUMP="gdas"
+
+# Get OBS for ENKFGDAS run from GDAS
+oCDUMP=${RUN}


so we don't export oCDUMP if the run if gdas?

Do we need export oCDUMP here? Both rCDUMP and oCDUMP are used just locally.

sorry I misread this line, but if they are only used internally then why export rCDUMP above on line 13?

I reused this by following the old version jobs/JGLOBAL_FORECAST for the GFS cycle to get the restart conditions. But I just checked and found they were modified as below:

# Restart conditions for GFS cycle come from GDAS rCDUMP=${RUN} export rCDUMP="${RUN/gfs/gdas}"

Do I need to make the corresponding changes?

CoryMartin-NOAA · 2024-04-03T19:14:08Z

+RUN=${RUN} YMD=${PDY} HH=${cyc} generate_com -rx COM_OBS_OUT:COM_OBS_TMPL

-RUN=${GDUMP} YMD=${gPDY} HH=${gcyc} generate_com -rx \
+RUN=${rCDUMP} YMD=${gPDY} HH=${gcyc} generate_com -rx \


shouldn't this be GDUMP?

The rCDUMP=${RUN} is defined locally for the GFS cycle to get the restart conditions from GDAS. For the GDAS cycle, rCDUMP is the same as GDUMP.

CoryMartin-NOAA · 2024-04-03T19:14:37Z

+RUN=${rCDUMP} YMD=${gPDY} HH=${gcyc} generate_com -rx \
    COM_ATMOS_RESTART_PREV:COM_ATMOS_RESTART_TMPL

+mkdir -m 775 -p "${COM_OBS_OUT}"


do we know for sure COM_OBS_IN exists already?

Yes, it is created in the gdasprep job.

CoryMartin-NOAA · 2024-04-03T19:15:23Z

    COM_RTOFS_TMPL=$(compath.py "${envir}/${WAVECUR_DID}/${rtofs_ver}")
 else
-    COM_OBS_TMPL='${ROTDIR}/${RUN}.${YMD}/${HH}/obs'
+    COM_OBS_TMPL='${ROTDIR}/${RUN}.${YMD}/${HH}/obs/${MEMDIR}'


This is going to do this for all ob types, isn't it, is this okay @aerorahul ?

Pinging @aerorahul again. Are we okay with creating a $MEMDIR for all observations? Will this work as intended out of the box for the atmosphere @jiaruidong2017 ? I don't know if it is preferred to have 80 largely identical sets of directories or 1 directory for most obs and 80 directories with just snow in them?

If we want that the COM_OBS points only to the gdas directory and without memdir, we can simply add MEMDIR="" where the COM_OBS is defined in all other jobs. An example is below:

RUN=${CDUMP} MEMDIR="" YMD=${PDY} HH=${cyc} declare_from_tmpl -rx COM_OBS:COM_OBS_TMPL

This PR updates several configures to prepare observations for snow DA updates to the ensemble members. Because the snow observations are fetched from the COM_OBS in the gdas deterministic analysis and are saved into the COM_OBS in the enkfgdas analysis for the enkfgdas run, we have to define two different COM_OBS directories (COMIN_OBS vs COMOUT_OBS) and two obs basename heads (OPREFIX_IN vs OPREFIX_OUT) to identify the enkf analysis from the gdas deterministic analysis. This PR will contribute to NOAA-EMC/global-workflow#2416. --------- Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>

jiaruidong2017 · 2024-04-16T21:50:33Z

@WalterKolczynski-NOAA I have updated the GDASApp to the current commit hash including the required changes by this PR. This PR is ready to proceed for a review. Thanks.

WalterKolczynski-NOAA · 2024-05-03T02:35:24Z

@jiaruidong2017 Sorry, I missed that this one was ready and kept skipping over it because of the blocked label. Please merge in develop and resolve any conflicts and we'll run it through CI.

CoryMartin-NOAA · 2024-05-03T12:33:11Z

Hold off on that, I need to chat with @jiaruidong2017 but I think we are going to end up closing this PR due to a proposed science change.

jiaruidong2017 · 2024-05-03T12:34:26Z

@WalterKolczynski-NOAA Because the GDASApp's PR was reverted, please hold up this PR until the GDASApp's PR is re-merged. Thanks.

CoryMartin-NOAA · 2024-05-08T17:32:14Z

We are going to try another direction for the ensemble update, and I'll close this PR.

This PR updates several configures to prepare observations for snow DA updates to the ensemble members. Because the snow observations are fetched from the COM_OBS in the gdas deterministic analysis and are saved into the COM_OBS in the enkfgdas analysis for the enkfgdas run, we have to define two different COM_OBS directories (COMIN_OBS vs COMOUT_OBS) and two obs basename heads (OPREFIX_IN vs OPREFIX_OUT) to identify the enkf analysis from the gdas deterministic analysis. This PR will contribute to NOAA-EMC/global-workflow#2416. --------- Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>

jiaruidong2017 added 2 commits March 18, 2024 18:01

Add the initial configure to prepare obs for the ensemble members.

1745619

Merge branch 'develop' into feature/prepsnowobs_ens

5fef5ad

WalterKolczynski-NOAA approved these changes Mar 27, 2024

View reviewed changes

jiaruidong2017 mentioned this pull request Mar 27, 2024

Prepare observations for snow DA updates to the ensemble members NOAA-EMC/GDASApp#998

Merged

WalterKolczynski-NOAA added the blocked Issue is currently being blocked by another issue. Include blocking issue # in description label Mar 27, 2024

jiaruidong2017 added 2 commits March 28, 2024 17:34

Merge branch 'develop' into feature/prepsnowobs_ens

b63a0e3

Merge branch 'develop' into feature/prepsnowobs_ens

fa05e5a

CoryMartin-NOAA reviewed Apr 1, 2024

View reviewed changes

Merge branch 'develop' into feature/prepsnowobs_ens

146e861

CoryMartin-NOAA reviewed Apr 3, 2024

View reviewed changes

jiaruidong2017 added 5 commits April 6, 2024 11:24

Change COM_OBS_IN to COMIN_OBS and change COM_OBS_OUT to COMOUT_OBS.

8ef5104

Merge branch 'develop' into feature/prepsnowobs_ens

6cc76e4

Merge branch 'develop' into feature/prepsnowobs_ens

a0f6ed2

Change generate_com to declare_from_tmpl.

15bbd45

Merge branch 'develop' into feature/prepsnowobs_ens

009767b

jiaruidong2017 added 2 commits April 16, 2024 16:47

Merge branch 'develop' into feature/prepsnowobs_ens

fc7cc84

Update the GDASApp commit hash.

2557cb7

Merge branch 'develop' into feature/prepsnowobs_ens

8b986b0

WalterKolczynski-NOAA removed the blocked Issue is currently being blocked by another issue. Include blocking issue # in description label May 3, 2024

CoryMartin-NOAA closed this May 8, 2024

Conversation

jiaruidong2017 commented Mar 20, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Type of change

Change characteristics

How has this been tested?

Checklist

Uh oh!

WalterKolczynski-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

WalterKolczynski-NOAA commented Mar 27, 2024

Uh oh!

jiaruidong2017 commented Mar 27, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiaruidong2017 Apr 6, 2024 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

jiaruidong2017 commented Apr 16, 2024

Uh oh!

WalterKolczynski-NOAA commented May 3, 2024

Uh oh!

CoryMartin-NOAA commented May 3, 2024

Uh oh!

jiaruidong2017 commented May 3, 2024

Uh oh!

CoryMartin-NOAA commented May 8, 2024

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jiaruidong2017 commented Mar 20, 2024 •

edited

Loading

jiaruidong2017 commented Mar 27, 2024 •

edited

Loading

jiaruidong2017 Apr 6, 2024 •

edited

Loading