Skip to content

Add ability for JEDI-to-FV3 increment converter to process ensembles#1022

Merged
CoryMartin-NOAA merged 27 commits into
developfrom
feature/lgetkf-jediinc2fv3
Apr 11, 2024
Merged

Add ability for JEDI-to-FV3 increment converter to process ensembles#1022
CoryMartin-NOAA merged 27 commits into
developfrom
feature/lgetkf-jediinc2fv3

Conversation

@DavidNew-NOAA
Copy link
Copy Markdown
Collaborator

@DavidNew-NOAA DavidNew-NOAA commented Apr 4, 2024

Major changes to JEDI-to-FV3 increment converter:

  1. Allow processing of ensembles using the same YAML template format that the FV3-JEDI LGETKF app uses. Alternatively, a list of background/increments can be specified.

  2. Fix incorrect calculation of FV3 increment, for variables other than delp and hydrostatic_delz, related to two issues:
    a. When adding the JEDI increment to the background to get the analysis, some mixing-ratio-type variables become
    negative and are set to zero by FV3-JEDI. However, when we then subtract the analysis and background after the
    variable change to get delp and hydrostatic_delz, we don't get the same increment that we started with for those
    mixing-ratio fields.
    b. For ensemble DA, the deterministic background and JEDI increment are not the same resolution, so the background
    and analysis discussed above must be interpolated to half resolution before being subtracted but after the JEDI
    increment is interpolated to full resolution and added to the background. Because of these two interpolations, similar
    to above, we don't get the same increment as we started with.

I now use Atlas to manually set all variables other than delp and hydrostatic_delz back to their original values, which solved both issues.

Additionally, I make the following other changes:

  1. Rename fv3jedi_fv3inc.yaml to fv3jedi_fv3inc_variational.yaml and rewrite according to the new YAML format.
  2. Rewrite the gdasapp_test_fv3jedi_fv3inc CTest input YAML according to the new YAML format.
  3. Create a new fv3jedi_fv3inc_lgetkf.yaml for an eventual use in converting the ensemble job increment in Global Workflow.

Testing these new features yields the following results:

  1. For delp and hydrostatic_delz in ensemble DA, the FV3 increment produced by the deterministic variational analysis in Global Workflow (branch feature/jediinc2fv3) is zero diff with develop, which has the previous version of the OOPS-based increment converter. I tested the other increment variables by writing the FV3 increment out to the native cubed sphere and compared with the JEDI increment. They are zero diff.
  2. The gdasapp_test_fv3jedi_fv3inc passes with the same test reference used in develop.
  3. Using a special Global Workflow branch to test the lgetkf analysis, the YAML templating works, and the resulting FV3 increment variables, delp and hydrostatic_delz, differ from the one values produced by develop (using jediinc2fv3.py) within a reasonable relative error, similar to that when I first tested the increment converter against jediinc2fv3.py with the variational analysis.

@DavidNew-NOAA DavidNew-NOAA marked this pull request as ready for review April 4, 2024 23:55
@DavidNew-NOAA
Copy link
Copy Markdown
Collaborator Author

Actually I have one more idea for a testing strategy, then will re-open...

@DavidNew-NOAA
Copy link
Copy Markdown
Collaborator Author

After a ton of debugging and testing, I'm finally re-opening this PR

@CoryMartin-NOAA CoryMartin-NOAA merged commit 005adcd into develop Apr 11, 2024
@CoryMartin-NOAA CoryMartin-NOAA deleted the feature/lgetkf-jediinc2fv3 branch April 11, 2024 13:26
danholdaway added a commit that referenced this pull request Apr 15, 2024
* upstream/develop:
  remove seviri from gdas_prototype_3d yaml (#1043)
  Fix GW jjob tests for upcoming GW PR #2420 (#1041)
  Fix test output for fv3jedi_fv3inc.h (#1039)
  Run g-w linker script before ctest for prepoceanobs task (#1034)
  Update femps and fv3-jedi-lm (#1036)
  Add ability for JEDI-to-FV3 increment converter to process ensembles (#1022)
  Add AVHRR/NOAA-15/18/19 assimilation to end-to-end GDASApp validation (#997)
  Catch error when trying to copy missing obs files from DATA to ROTDIR in prepoceanobs (#1028)
DavidNew-NOAA pushed a commit that referenced this pull request Jan 6, 2026
This PR adds the `state variables to inverse` block to
-
`model/atmosphere/atmosphere_background_error_hybrid_gsibec_bump.yaml.j2`
- `model/atmosphere/atmosphere_background_error_static_gsibec.yaml.j2`

Updates to saber at
[dce89ba](JCSDA/saber@dce89ba)
and fv3-jedi at
[7e8337d](JCSDA/fv3-jedi@7e8337d)
require addition of a state variables to inverse block in GSIBEC yamls.

See saber PR [#1022](JCSDA-internal/saber#1022)
and fv3-jedi PR
[#1350](JCSDA-internal/fv3-jedi#1350) for
details.

Resolves #92

The changes in this PR have been tested in GDASApp
`feature/stable-nightly` and found to work.
DavidNew-NOAA added a commit that referenced this pull request Jan 16, 2026
…1022)

Major changes to JEDI-to-FV3 increment converter:

1. Allow processing of ensembles using the same YAML template format
that the FV3-JEDI LGETKF app uses. Alternatively, a list of
background/increments can be specified.

2. Fix incorrect calculation of FV3 increment, for variables other than
delp and hydrostatic_delz, related to two issues:
a. When adding the JEDI increment to the background to get the analysis,
some mixing-ratio-type variables become
negative and are set to zero by FV3-JEDI. However, when we then subtract
the analysis and background after the
variable change to get delp and hydrostatic_delz, we don't get the same
increment that we started with for those
      mixing-ratio fields. 
b. For ensemble DA, the deterministic background and JEDI increment are
not the same resolution, so the background
and analysis discussed above must be interpolated to half resolution
before being subtracted but after the JEDI
increment is interpolated to full resolution and added to the
background. Because of these two interpolations, similar
      to above, we don't get the same increment as we started with.

I now use Atlas to manually set all variables other than delp and
hydrostatic_delz back to their original values, which solved both
issues.

Additionally, I make the following other changes:

1. Rename fv3jedi_fv3inc.yaml to fv3jedi_fv3inc_variational.yaml and
rewrite according to the new YAML format.
4. Rewrite the gdasapp_test_fv3jedi_fv3inc CTest input YAML according to
the new YAML format.
5. Create a new fv3jedi_fv3inc_lgetkf.yaml for an eventual use in
converting the ensemble job increment in Global Workflow.

Testing these new features yields the following results:

1. For delp and hydrostatic_delz in ensemble DA, the FV3 increment
produced by the deterministic variational analysis in Global Workflow
(branch feature/jediinc2fv3) is zero diff with develop, which has the
previous version of the OOPS-based increment converter. I tested the
other increment variables by writing the FV3 increment out to the native
cubed sphere and compared with the JEDI increment. They are zero diff.
2. The gdasapp_test_fv3jedi_fv3inc passes with the same test reference
used in develop.
3. Using a special Global Workflow branch to test the lgetkf analysis,
the YAML templating works, and the resulting FV3 increment variables,
delp and hydrostatic_delz, differ from the one values produced by
develop (using jediinc2fv3.py) within a reasonable relative error,
similar to that when I first tested the increment converter against
jediinc2fv3.py with the variational analysis.
RussTreadon-NOAA added a commit that referenced this pull request Jan 16, 2026
This PR adds the `state variables to inverse` block to
-
`model/atmosphere/atmosphere_background_error_hybrid_gsibec_bump.yaml.j2`
- `model/atmosphere/atmosphere_background_error_static_gsibec.yaml.j2`

Updates to saber at
[dce89ba](JCSDA/saber@dce89ba)
and fv3-jedi at
[7e8337d](JCSDA/fv3-jedi@7e8337d)
require addition of a state variables to inverse block in GSIBEC yamls.

See saber PR [#1022](JCSDA-internal/saber#1022)
and fv3-jedi PR
[#1350](JCSDA-internal/fv3-jedi#1350) for
details.

Resolves #92

The changes in this PR have been tested in GDASApp
`feature/stable-nightly` and found to work.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants