Skip to content

Feature/addb2i - added bufr to ioda converters (from GDASAPP) and testing#22

Merged
guillaumevernieres merged 3 commits into
mainfrom
feature/addb2i
Mar 20, 2025
Merged

Feature/addb2i - added bufr to ioda converters (from GDASAPP) and testing#22
guillaumevernieres merged 3 commits into
mainfrom
feature/addb2i

Conversation

@givelberg
Copy link
Copy Markdown
Contributor

@givelberg givelberg commented Mar 20, 2025

This was tested on orion and hera.
TODO: testing on wcoss (when my account problem gets resolved)

Edited by guillaume

Copy link
Copy Markdown
Contributor

@guillaumevernieres guillaumevernieres left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @givelberg .
We'll probably need to figure out how to test this inside the GitHub runner at some point, a problem for another day!

@guillaumevernieres guillaumevernieres merged commit 63ef80c into main Mar 20, 2025
@guillaumevernieres guillaumevernieres deleted the feature/addb2i branch March 20, 2025 12:17
@apchoiCMD
Copy link
Copy Markdown
Contributor

Tested on Hera

[Mindo.Choi@hfe06 obsforge-utils]$ ctest
Test project /scratch1/NCEPDEV/da/Mindo.Choi/obsForge/build/obsforge-utils
      Start  1: test_b2i_bufr2ioda_insitu_profile_argo
 1/21 Test  #1: test_b2i_bufr2ioda_insitu_profile_argo .......   Passed   12.14 sec
      Start  2: test_b2i_bufr2ioda_insitu_profile_bathy
 2/21 Test  #2: test_b2i_bufr2ioda_insitu_profile_bathy ......   Passed    2.46 sec
      Start  3: test_b2i_bufr2ioda_insitu_profile_glider
 3/21 Test  #3: test_b2i_bufr2ioda_insitu_profile_glider .....   Passed    4.33 sec
      Start  4: test_b2i_bufr2ioda_insitu_profile_tesac
 4/21 Test  #4: test_b2i_bufr2ioda_insitu_profile_tesac ......   Passed    7.09 sec
      Start  5: test_b2i_bufr2ioda_insitu_profile_tropical
 5/21 Test  #5: test_b2i_bufr2ioda_insitu_profile_tropical ...   Passed    2.89 sec
      Start  6: test_b2i_bufr2ioda_insitu_profile_xbtctd
 6/21 Test  #6: test_b2i_bufr2ioda_insitu_profile_xbtctd .....   Passed    2.31 sec
      Start  7: test_b2i_bufr2ioda_insitu_surface_drifter
 7/21 Test  #7: test_b2i_bufr2ioda_insitu_surface_drifter ....   Passed    2.49 sec
      Start  8: test_b2i_bufr2ioda_insitu_surface_trkob
 8/21 Test  #8: test_b2i_bufr2ioda_insitu_surface_trkob ......   Passed    2.53 sec
      Start  9: test_obsforge_util_coding_norms
 9/21 Test  #9: test_obsforge_util_coding_norms ..............   Passed    1.73 sec
      Start 10: test_obsforge_util_prepdata
10/21 Test #10: test_obsforge_util_prepdata ..................   Passed    3.34 sec
      Start 11: test_obsforge_util_rads2ioda
11/21 Test #11: test_obsforge_util_rads2ioda .................   Passed    0.37 sec
      Start 12: test_obsforge_util_ghrsst2ioda
12/21 Test #12: test_obsforge_util_ghrsst2ioda ...............   Passed    0.17 sec
      Start 13: test_obsforge_util_rtofstmp
13/21 Test #13: test_obsforge_util_rtofstmp ..................   Passed    0.93 sec
      Start 14: test_obsforge_util_rtofssal
14/21 Test #14: test_obsforge_util_rtofssal ..................   Passed    0.91 sec
      Start 15: test_obsforge_util_smap2ioda
15/21 Test #15: test_obsforge_util_smap2ioda .................   Passed    0.16 sec
      Start 16: test_obsforge_util_smos2ioda
16/21 Test #16: test_obsforge_util_smos2ioda .................   Passed    0.19 sec
      Start 17: test_obsforge_util_viirsaod2ioda
17/21 Test #17: test_obsforge_util_viirsaod2ioda .............   Passed    0.19 sec
      Start 18: test_obsforge_util_icecamsr2ioda
18/21 Test #18: test_obsforge_util_icecamsr2ioda .............   Passed    0.18 sec
      Start 19: test_obsforge_util_icecmirs2ioda
19/21 Test #19: test_obsforge_util_icecmirs2ioda .............   Passed    0.18 sec
      Start 20: test_obsforge_util_icecjpssrr2ioda
20/21 Test #20: test_obsforge_util_icecjpssrr2ioda ...........   Passed    0.15 sec
      Start 21: test_obsforge_util_insituall2ioda
21/21 Test #21: test_obsforge_util_insituall2ioda ............   Passed    0.15 sec

100% tests passed, 0 tests failed out of 21

Label Time Summary:
obsforge-utils    =   8.66 sec*proc (13 tests)
script            =   8.66 sec*proc (13 tests)

Total Test time (real) =  45.00 sec

@ilianagenkova
Copy link
Copy Markdown

Builds and tests ok on WCOSS/Cactus:
image

@apchoiCMD
Copy link
Copy Markdown
Contributor

Builds and tests ok on WCOSS/Cactus: image

@ilianagenkova you need to git pull and rebuild them again, your build doesn't include @givelberg stuff, please see above on my build.

@givelberg
Copy link
Copy Markdown
Contributor Author

Builds and tests ok on WCOSS/Cactus:

@ilianagenkova
since you are building on WCOSS, you are not seeing the b2i tests at all.

b2i tests currently require bufr files to be staged
they are staged in specific locations on orion and hera,

My CMakelists.txt file generates the b2i test only if it can find the staged bufr files.

This is what I wrote in the TODO description of this pr

@ilianagenkova
Copy link
Copy Markdown

OK, I'll step back.

CoryMartin-NOAA added a commit that referenced this pull request Oct 21, 2025
…sts (#12)

1. Simplified ctest structure.
My initial version is very stupid and inefficient!
e.g. initial ctest list for test_obsforge_satobs_satwnd_amv_seviri looks
like the following:
```
  Test  #38: test_obsforge_satobs_satwnd_amv_seviri_m8_bufr2netcdf
  Test  #39: test_obsforge_satobs_satwnd_amv_seviri_m8_bufr2netcdf_mpi4
  Test  #40: test_obsforge_satobs_satwnd_amv_seviri_m8_script2netcdf
  Test  #41: test_obsforge_satobs_satwnd_amv_seviri_m8_script2netcdf_mpi4
  Test  #42: test_obsforge_satobs_satwnd_amv_seviri_m8_bufr4backend
  Test  #43: test_obsforge_satobs_satwnd_amv_seviri_m8_bufr4backend_mpi4
  Test  #44: test_obsforge_satobs_satwnd_amv_seviri_m8_script4backend
  Test  #45: test_obsforge_satobs_satwnd_amv_seviri_m8_script4backend_mpi4
  Test  #46: test_obsforge_satobs_satwnd_amv_seviri_m9_bufr2netcdf
  Test  #47: test_obsforge_satobs_satwnd_amv_seviri_m9_bufr2netcdf_mpi4
  Test  #48: test_obsforge_satobs_satwnd_amv_seviri_m9_script2netcdf
  Test  #49: test_obsforge_satobs_satwnd_amv_seviri_m9_script2netcdf_mpi4
  Test  #50: test_obsforge_satobs_satwnd_amv_seviri_m9_bufr4backend
  Test  #51: test_obsforge_satobs_satwnd_amv_seviri_m9_bufr4backend_mpi4
  Test  #52: test_obsforge_satobs_satwnd_amv_seviri_m9_script4backend
  Test  #53: test_obsforge_satobs_satwnd_amv_seviri_m9_script4backend_mpi4
  Test  #54: test_obsforge_satobs_satwnd_amv_seviri_m10_bufr2netcdf
  Test  #55: test_obsforge_satobs_satwnd_amv_seviri_m10_bufr2netcdf_mpi4
  Test  #56: test_obsforge_satobs_satwnd_amv_seviri_m10_script2netcdf
  Test  #57: test_obsforge_satobs_satwnd_amv_seviri_m10_script2netcdf_mpi4
  Test  #58: test_obsforge_satobs_satwnd_amv_seviri_m10_bufr4backend
  Test  #59: test_obsforge_satobs_satwnd_amv_seviri_m10_bufr4backend_mpi4
  Test  #60: test_obsforge_satobs_satwnd_amv_seviri_m10_script4backend
  Test  #61: test_obsforge_satobs_satwnd_amv_seviri_m10_script4backend_mpi4
  Test  #62: test_obsforge_satobs_satwnd_amv_seviri_m11_bufr2netcdf
  Test  #63: test_obsforge_satobs_satwnd_amv_seviri_m11_bufr2netcdf_mpi4
  Test  #64: test_obsforge_satobs_satwnd_amv_seviri_m11_script2netcdf
  Test  #65: test_obsforge_satobs_satwnd_amv_seviri_m11_script2netcdf_mpi4
  Test  #66: test_obsforge_satobs_satwnd_amv_seviri_m11_bufr4backend
  Test  #67: test_obsforge_satobs_satwnd_amv_seviri_m11_bufr4backend_mpi4
  Test  #68: test_obsforge_satobs_satwnd_amv_seviri_m11_script4backend
  Test  #69: test_obsforge_satobs_satwnd_amv_seviri_m11_script4backend_mpi4
```

The simplified one looks like the following for satwnd_amv_seviri:
``` 
      Start 22: test_obsforge_satobs_satwnd_amv_seviri_bufr2netcdf
 7/38 Test #22: test_obsforge_satobs_satwnd_amv_seviri_bufr2netcdf ...........   Passed   16.69 sec
      Start 23: test_obsforge_satobs_satwnd_amv_seviri_bufr2netcdf_mpi4
 8/38 Test #23: test_obsforge_satobs_satwnd_amv_seviri_bufr2netcdf_mpi4 ......   Passed    9.47 sec
      Start 24: test_obsforge_satobs_satwnd_amv_seviri_script2netcdf
 9/38 Test #24: test_obsforge_satobs_satwnd_amv_seviri_script2netcdf .........   Passed   18.98 sec
      Start 25: test_obsforge_satobs_satwnd_amv_seviri_script2netcdf_mpi4
10/38 Test #25: test_obsforge_satobs_satwnd_amv_seviri_script2netcdf_mpi4 ....   Passed   13.31 sec
      Start 26: test_obsforge_satobs_satwnd_amv_seviri_bufr4backend
11/38 Test #26: test_obsforge_satobs_satwnd_amv_seviri_bufr4backend ..........   Passed   18.95 sec
      Start 27: test_obsforge_satobs_satwnd_amv_seviri_bufr4backend_mpi4
12/38 Test #27: test_obsforge_satobs_satwnd_amv_seviri_bufr4backend_mpi4 .....   Passed   11.18 sec
      Start 28: test_obsforge_satobs_satwnd_amv_seviri_script4backend
13/38 Test #28: test_obsforge_satobs_satwnd_amv_seviri_script4backend ........   Passed   22.38 sec
      Start 29: test_obsforge_satobs_satwnd_amv_seviri_script4backend_mpi4
14/38 Test #29: test_obsforge_satobs_satwnd_amv_seviri_script4backend_mpi4 ...   Passed   16.85 sec
```
The test structure is simplified in that the test for each data type,
and each configuration only needs to be tested once because the
`bufr_comp.sh` has been modified to handle comparisons of output files
from various satellite platforms for the same sensor type. (See
[bufr-query PR #53](NOAA-EMC/bufr-query#53))

2. Enable IASI tests
    - add tests in CMakeList.txt
    - add test references (testoutput) in obsForge-0.0.0.tgz on EMCRZDM
    - add test configuration YAMLs (testinput)
- mapping file and python configuration YAMLs are added in [SPOC
PR#25](NOAA-EMC/spoc#25)

---------

Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Migrate the observation pre-processing to obsForge

4 participants