Add Python Script to Translate Increments by CoryMartin-NOAA · Pull Request #2 · NOAA-EMC/GDASApp

CoryMartin-NOAA · 2022-03-17T20:06:42Z

This PR:

Adds a python script that can 'translate' FV3-JEDI LatLon increments to current FV3 format
Makes the ctest run in CI only with pull requests
Adds a capability to download 'fix files' from EMC FTP server
Adds a unit test to test the increment script

Note: I have NOT tested the output file yet with the model, but it 'looks' like it should work.

aerorahul

looks good.

CoryMartin-NOAA · 2022-03-17T20:12:37Z

looks good.

Except the CI is failing :-)

guillaumevernieres · 2022-03-17T20:28:16Z

+  EXPECTED_HASH SHA256=${SHA}
+  )
+# Extract downloaded tarball.
+file(ARCHIVE_EXTRACT INPUT ${TAR})


Need to change the cmake min version to something more recent. 3.16.3 is too old for the option above, 3.20.0 seems fine.

guillaumevernieres

👍

aerorahul · 2022-03-21T14:53:41Z

+    assert os.path.exists(FV3JEDIinc), f"File {FV3JEDIinc} not found."
+    assert os.path.exists(FV3JEDIinc), f"File {FV3JEDIinc} not found."


use try instead of assert. Assertions are great for development, but when too many assertions are used, they can slow down the code significantly.
Besides, this is exactly what try is for and then catch the Exception of FileNotFoundError.

…DASApp into feature/translate_inc

aerorahul

looks good.

Only tested with the coupled UFS and the marine DA tasks: ``` Test project /scratch1/NCEPDEV/stmp2/Guillaume.Vernieres/runs/prs/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci Test #1: C48mx500_3DVarAOWCDA Test #2: C96_atmaerosnowDA Test #3: C96C48_ufs_hybatmDA Test #4: C48mx500_3DVarAOWCDA_gdasfcst_202103241200 Test #5: C48mx500_3DVarAOWCDA_gdasprepoceanobs_202103241800 Test #6: C48mx500_3DVarAOWCDA_gdasocnanalprep_202103241800 Test #7: C48mx500_3DVarAOWCDA_gdasocnanalbmat_202103241800 Test #8: C48mx500_3DVarAOWCDA_gdasocnanalrun_202103241800 Test #9: C48mx500_3DVarAOWCDA_gdasocnanalchkpt_202103241800 Test #10: C48mx500_3DVarAOWCDA_gdasocnanalpost_202103241800 Total Tests: 10 ```

**What was done:** - reorganization/code tidy to facilitate the addition of a GFSv17 prototype ctest - exclude the new ctest by default, the user will have to run ```cmake -DRUNGWCI=ON .``` on an old build to configure the new ctest He're the current list: ``` Test project /scratch1/NCEPDEV/stmp2/Guillaume.Vernieres/runs/prs/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci Test #1: WCDA-3DVAR-C48mx500 Test #2: WCDA-3DVAR-C48mx500_gdasfcst_202103241200 Test #3: WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800 Test #4: WCDA-3DVAR-C48mx500_gdasocnanalprep_202103241800 Test #5: WCDA-3DVAR-C48mx500_gdasocnanalbmat_202103241800 Test #6: WCDA-3DVAR-C48mx500_gdasocnanalrun_202103241800 Test #7: WCDA-3DVAR-C48mx500_gdasocnanalchkpt_202103241800 Test #8: WCDA-3DVAR-C48mx500_gdasocnanalpost_202103241800 Test #9: Aero-Snow-3DVAR-C96 Test #10: Aero-Snow-3DVAR-C96_gdasfcst_202112201200 Test #11: Atm-hyb-C96C48 Test #12: Atm-hyb-C96C48_gdasfcst_202402231800 Test #13: GFSv17-3DVAR-C384mx025 Test #14: GFSv17-3DVAR-C384mx025_gdasfcst_202106300000 Test #15: GFSv17-3DVAR-C384mx025_gdasprepoceanobs_202106300600 Test #16: GFSv17-3DVAR-C384mx025_gdasocnanalprep_202106300600 Test #17: GFSv17-3DVAR-C384mx025_gdasocnanalbmat_202106300600 Test #18: GFSv17-3DVAR-C384mx025_gdasocnanalrun_202106300600 Test #19: GFSv17-3DVAR-C384mx025_gdasocnanalchkpt_202106300600 Test #20: GFSv17-3DVAR-C384mx025_gdasocnanalpost_202106300600 Test #21: GFSv17-3DVAR-C384mx025_gdasocnanalvrfy_202106300600 Test #22: GFSv17-3DVAR-C384mx025_gdasprep_202106300600 Test #23: GFSv17-3DVAR-C384mx025_gdasanal_202106300600 Total Tests: 23 ``` --------- Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>

minor adjustments required by g-w JEDI ATM CI

* Add pycodestyle install * Make ctest verbose in CI * Add netCDF4 to CI * No need to run pynorms so many times * update min cmake version

Only tested with the coupled UFS and the marine DA tasks: ``` Test project /scratch1/NCEPDEV/stmp2/Guillaume.Vernieres/runs/prs/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci Test #1: C48mx500_3DVarAOWCDA Test #2: C96_atmaerosnowDA Test #3: C96C48_ufs_hybatmDA Test #4: C48mx500_3DVarAOWCDA_gdasfcst_202103241200 Test #5: C48mx500_3DVarAOWCDA_gdasprepoceanobs_202103241800 Test #6: C48mx500_3DVarAOWCDA_gdasocnanalprep_202103241800 Test #7: C48mx500_3DVarAOWCDA_gdasocnanalbmat_202103241800 Test #8: C48mx500_3DVarAOWCDA_gdasocnanalrun_202103241800 Test #9: C48mx500_3DVarAOWCDA_gdasocnanalchkpt_202103241800 Test #10: C48mx500_3DVarAOWCDA_gdasocnanalpost_202103241800 Total Tests: 10 ```

**What was done:** - reorganization/code tidy to facilitate the addition of a GFSv17 prototype ctest - exclude the new ctest by default, the user will have to run ```cmake -DRUNGWCI=ON .``` on an old build to configure the new ctest He're the current list: ``` Test project /scratch1/NCEPDEV/stmp2/Guillaume.Vernieres/runs/prs/global-workflow/sorc/gdas.cd/build/gdas/test/gw-ci Test #1: WCDA-3DVAR-C48mx500 Test #2: WCDA-3DVAR-C48mx500_gdasfcst_202103241200 Test #3: WCDA-3DVAR-C48mx500_gdasprepoceanobs_202103241800 Test #4: WCDA-3DVAR-C48mx500_gdasocnanalprep_202103241800 Test #5: WCDA-3DVAR-C48mx500_gdasocnanalbmat_202103241800 Test #6: WCDA-3DVAR-C48mx500_gdasocnanalrun_202103241800 Test #7: WCDA-3DVAR-C48mx500_gdasocnanalchkpt_202103241800 Test #8: WCDA-3DVAR-C48mx500_gdasocnanalpost_202103241800 Test #9: Aero-Snow-3DVAR-C96 Test #10: Aero-Snow-3DVAR-C96_gdasfcst_202112201200 Test #11: Atm-hyb-C96C48 Test #12: Atm-hyb-C96C48_gdasfcst_202402231800 Test #13: GFSv17-3DVAR-C384mx025 Test #14: GFSv17-3DVAR-C384mx025_gdasfcst_202106300000 Test #15: GFSv17-3DVAR-C384mx025_gdasprepoceanobs_202106300600 Test #16: GFSv17-3DVAR-C384mx025_gdasocnanalprep_202106300600 Test #17: GFSv17-3DVAR-C384mx025_gdasocnanalbmat_202106300600 Test #18: GFSv17-3DVAR-C384mx025_gdasocnanalrun_202106300600 Test #19: GFSv17-3DVAR-C384mx025_gdasocnanalchkpt_202106300600 Test #20: GFSv17-3DVAR-C384mx025_gdasocnanalpost_202106300600 Test #21: GFSv17-3DVAR-C384mx025_gdasocnanalvrfy_202106300600 Test #22: GFSv17-3DVAR-C384mx025_gdasprep_202106300600 Test #23: GFSv17-3DVAR-C384mx025_gdasanal_202106300600 Total Tests: 23 ``` --------- Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>

minor adjustments required by g-w JEDI ATM CI

CoryMartin-NOAA added 5 commits March 17, 2022 13:42

Save commit of partial work

70f6c40

Working converter

a768067

Coding norms

c75d50d

Attempt at gathering test data and new unit test

ed3f13e

Working ctest

313eb87

CoryMartin-NOAA self-assigned this Mar 17, 2022

CoryMartin-NOAA requested review from RussTreadon-NOAA, aerorahul and guillaumevernieres March 17, 2022 20:07

Add pycodestyle install

13528e0

aerorahul previously approved these changes Mar 17, 2022

View reviewed changes

Make ctest verbose in CI

69dda87

CoryMartin-NOAA dismissed aerorahul’s stale review via 69dda87 March 17, 2022 20:13

CoryMartin-NOAA added 2 commits March 17, 2022 20:16

Add netCDF4 to CI

91e9879

No need to run pynorms so many times

5498453

CoryMartin-NOAA requested a review from aerorahul March 17, 2022 20:19

guillaumevernieres reviewed Mar 17, 2022

View reviewed changes

update min cmake version

a8699d2

guillaumevernieres previously approved these changes Mar 17, 2022

View reviewed changes

aerorahul reviewed Mar 21, 2022

View reviewed changes

remove assert, wrap everything in a try

46acaa6

CoryMartin-NOAA dismissed guillaumevernieres’s stale review via 46acaa6 March 21, 2022 19:02

CoryMartin-NOAA added 3 commits March 21, 2022 15:03

Merge branch 'develop' into feature/translate_inc

3d0fdb7

Fix funky indentation

4283ca1

Merge branch 'feature/translate_inc' of https://github.com/NOAA-EMC/G…

a382ac4

…DASApp into feature/translate_inc

CoryMartin-NOAA requested review from aerorahul and guillaumevernieres March 21, 2022 19:15

aerorahul previously approved these changes Mar 21, 2022

View reviewed changes

Update CMakeLists.txt

faa972f

CoryMartin-NOAA dismissed aerorahul’s stale review via faa972f March 21, 2022 19:36

guillaumevernieres approved these changes Mar 21, 2022

View reviewed changes

CoryMartin-NOAA requested a review from aerorahul March 22, 2022 18:19

CoryMartin-NOAA added the orion-RT Queue for automated testing on Orion label Mar 22, 2022

emcbot added orion-RT-Running Automated testing running on Orion and removed orion-RT Queue for automated testing on Orion labels Mar 22, 2022

CoryMartin-NOAA removed the orion-RT-Running Automated testing running on Orion label Mar 22, 2022

CoryMartin-NOAA merged commit 45d4de9 into develop Mar 23, 2022

CoryMartin-NOAA deleted the feature/translate_inc branch March 23, 2022 15:19

hyunchul386 mentioned this pull request Sep 13, 2022

Prepare JEDI increment so it can be used by MOM6 iau #121

Closed

3 tasks

ilianagenkova mentioned this pull request Mar 19, 2025

Prepare the marine obs for several retro test #1513

Closed

3 tasks

JohnSteffen-NOAA mentioned this pull request May 6, 2025

bufr2ioda fails on WCOSS2 #1655

Closed

DavidNew-NOAA pushed a commit that referenced this pull request Jan 6, 2026

Merge pull request #2 from RussTreadon-NOAA/feature/jedi_atm_ci

96efdf7

minor adjustments required by g-w JEDI ATM CI

DavidNew-NOAA pushed a commit that referenced this pull request Jan 16, 2026

Add Python Script to Translate Increments (#2)

4862ecf

* Add pycodestyle install * Make ctest verbose in CI * Add netCDF4 to CI * No need to run pynorms so many times * update min cmake version

RussTreadon-NOAA pushed a commit that referenced this pull request Jan 16, 2026

Merge pull request #2 from RussTreadon-NOAA/feature/jedi_atm_ci

f0f3eef

minor adjustments required by g-w JEDI ATM CI

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add Python Script to Translate Increments#2

Add Python Script to Translate Increments#2
CoryMartin-NOAA merged 15 commits into
developfrom
feature/translate_inc

CoryMartin-NOAA commented Mar 17, 2022

Uh oh!

aerorahul left a comment

Uh oh!

CoryMartin-NOAA commented Mar 17, 2022

Uh oh!

guillaumevernieres Mar 17, 2022

Uh oh!

guillaumevernieres left a comment

Uh oh!

aerorahul Mar 21, 2022

Uh oh!

aerorahul left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

		assert os.path.exists(FV3JEDIinc), f"File {FV3JEDIinc} not found."
		assert os.path.exists(FV3JEDIinc), f"File {FV3JEDIinc} not found."

Conversation

CoryMartin-NOAA commented Mar 17, 2022

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

CoryMartin-NOAA commented Mar 17, 2022

Uh oh!

guillaumevernieres Mar 17, 2022

Choose a reason for hiding this comment

Uh oh!

guillaumevernieres left a comment

Choose a reason for hiding this comment

Uh oh!

aerorahul Mar 21, 2022

Choose a reason for hiding this comment

Uh oh!

aerorahul left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants