Adds stubs for marine obs bufr converter job#68
Merged
Conversation
AndrewEichmann-NOAA
requested review from
apchoiCMD,
Copilot and
guillaumevernieres
AndrewEichmann-NOAA
changed the title
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR adds the necessary stubs and configuration to support a new marine bufr converter job. Key changes include:
- Introduction of a stub task class (MarineBufrObsPrep) in Python with basic logging in its methods.
- Addition of a new rocoto template, job scripts, and configuration entries for the marine bufr processing.
- Updates across YAML, shell, and XML files to integrate the new job into the workflow.
Reviewed Changes
Copilot reviewed 6 out of 6 changed files in this pull request and generated 1 comment.
Show a summary per file
| File | Description |
|---|---|
| ush/python/pyobsforge/task/marine_bufr_prepobs.py | Adds a new task class with stub methods for marine observations. |
| scripts/exobsforge_global_marine_bufr_dump.py | Provides a new driver script to run the marine bufr task. |
| parm/obsforge_rocoto_template.xml.j2 | Introduces a new rocoto task for the marine bufr dump. |
| parm/config.hera.yaml | Updates configuration with new keys for the marine bufr dump. |
| jobs/rocoto/marinebufrdump.sh | Adds a shell script to bootstrap the marine bufr dump job. |
| jobs/JOBSFORGE_GLOBAL_MARINE_BUFR_DUMP | Implements the job submission logic for the marine bufr dump. |
Comments suppressed due to low confidence (2)
parm/obsforge_rocoto_template.xml.j2:144
- [nitpick] Ensure consistent naming conventions for the marine bufr dump task across configuration and job scripts (e.g., 'gfs_marine_bufr_dump' in XML vs. 'marinebufrdump' in the configuration).
<task name="gfs_marine_bufr_dump" cycledefs="gfs" maxtries="&MAXTRIES;">
jobs/JOBSFORGE_GLOBAL_MARINE_BUFR_DUMP:33
- [nitpick] Clarify the origin and expected assignment of the 'pgmout' variable to ensure that its use in this file is intentional and robust against potential undefined behavior.
if [[ -e "${pgmout}" ]] ; then
guillaumevernieres
approved these changes
May 20, 2025
Contributor
guillaumevernieres
left a comment
guillaumevernieres
left a comment
There was a problem hiding this comment.
Seems fine. Thanks @AndrewEichmann-NOAA
apchoiCMD
approved these changes
May 20, 2025
CoryMartin-NOAA
added a commit
that referenced
this pull request
Oct 21, 2025
…sts (#12) 1. Simplified ctest structure. My initial version is very stupid and inefficient! e.g. initial ctest list for test_obsforge_satobs_satwnd_amv_seviri looks like the following: ``` Test #38: test_obsforge_satobs_satwnd_amv_seviri_m8_bufr2netcdf Test #39: test_obsforge_satobs_satwnd_amv_seviri_m8_bufr2netcdf_mpi4 Test #40: test_obsforge_satobs_satwnd_amv_seviri_m8_script2netcdf Test #41: test_obsforge_satobs_satwnd_amv_seviri_m8_script2netcdf_mpi4 Test #42: test_obsforge_satobs_satwnd_amv_seviri_m8_bufr4backend Test #43: test_obsforge_satobs_satwnd_amv_seviri_m8_bufr4backend_mpi4 Test #44: test_obsforge_satobs_satwnd_amv_seviri_m8_script4backend Test #45: test_obsforge_satobs_satwnd_amv_seviri_m8_script4backend_mpi4 Test #46: test_obsforge_satobs_satwnd_amv_seviri_m9_bufr2netcdf Test #47: test_obsforge_satobs_satwnd_amv_seviri_m9_bufr2netcdf_mpi4 Test #48: test_obsforge_satobs_satwnd_amv_seviri_m9_script2netcdf Test #49: test_obsforge_satobs_satwnd_amv_seviri_m9_script2netcdf_mpi4 Test #50: test_obsforge_satobs_satwnd_amv_seviri_m9_bufr4backend Test #51: test_obsforge_satobs_satwnd_amv_seviri_m9_bufr4backend_mpi4 Test #52: test_obsforge_satobs_satwnd_amv_seviri_m9_script4backend Test #53: test_obsforge_satobs_satwnd_amv_seviri_m9_script4backend_mpi4 Test #54: test_obsforge_satobs_satwnd_amv_seviri_m10_bufr2netcdf Test #55: test_obsforge_satobs_satwnd_amv_seviri_m10_bufr2netcdf_mpi4 Test #56: test_obsforge_satobs_satwnd_amv_seviri_m10_script2netcdf Test #57: test_obsforge_satobs_satwnd_amv_seviri_m10_script2netcdf_mpi4 Test #58: test_obsforge_satobs_satwnd_amv_seviri_m10_bufr4backend Test #59: test_obsforge_satobs_satwnd_amv_seviri_m10_bufr4backend_mpi4 Test #60: test_obsforge_satobs_satwnd_amv_seviri_m10_script4backend Test #61: test_obsforge_satobs_satwnd_amv_seviri_m10_script4backend_mpi4 Test #62: test_obsforge_satobs_satwnd_amv_seviri_m11_bufr2netcdf Test #63: test_obsforge_satobs_satwnd_amv_seviri_m11_bufr2netcdf_mpi4 Test #64: test_obsforge_satobs_satwnd_amv_seviri_m11_script2netcdf Test #65: test_obsforge_satobs_satwnd_amv_seviri_m11_script2netcdf_mpi4 Test #66: test_obsforge_satobs_satwnd_amv_seviri_m11_bufr4backend Test #67: test_obsforge_satobs_satwnd_amv_seviri_m11_bufr4backend_mpi4 Test #68: test_obsforge_satobs_satwnd_amv_seviri_m11_script4backend Test #69: test_obsforge_satobs_satwnd_amv_seviri_m11_script4backend_mpi4 ``` The simplified one looks like the following for satwnd_amv_seviri: ``` Start 22: test_obsforge_satobs_satwnd_amv_seviri_bufr2netcdf 7/38 Test #22: test_obsforge_satobs_satwnd_amv_seviri_bufr2netcdf ........... Passed 16.69 sec Start 23: test_obsforge_satobs_satwnd_amv_seviri_bufr2netcdf_mpi4 8/38 Test #23: test_obsforge_satobs_satwnd_amv_seviri_bufr2netcdf_mpi4 ...... Passed 9.47 sec Start 24: test_obsforge_satobs_satwnd_amv_seviri_script2netcdf 9/38 Test #24: test_obsforge_satobs_satwnd_amv_seviri_script2netcdf ......... Passed 18.98 sec Start 25: test_obsforge_satobs_satwnd_amv_seviri_script2netcdf_mpi4 10/38 Test #25: test_obsforge_satobs_satwnd_amv_seviri_script2netcdf_mpi4 .... Passed 13.31 sec Start 26: test_obsforge_satobs_satwnd_amv_seviri_bufr4backend 11/38 Test #26: test_obsforge_satobs_satwnd_amv_seviri_bufr4backend .......... Passed 18.95 sec Start 27: test_obsforge_satobs_satwnd_amv_seviri_bufr4backend_mpi4 12/38 Test #27: test_obsforge_satobs_satwnd_amv_seviri_bufr4backend_mpi4 ..... Passed 11.18 sec Start 28: test_obsforge_satobs_satwnd_amv_seviri_script4backend 13/38 Test #28: test_obsforge_satobs_satwnd_amv_seviri_script4backend ........ Passed 22.38 sec Start 29: test_obsforge_satobs_satwnd_amv_seviri_script4backend_mpi4 14/38 Test #29: test_obsforge_satobs_satwnd_amv_seviri_script4backend_mpi4 ... Passed 16.85 sec ``` The test structure is simplified in that the test for each data type, and each configuration only needs to be tested once because the `bufr_comp.sh` has been modified to handle comparisons of output files from various satellite platforms for the same sensor type. (See [bufr-query PR #53](NOAA-EMC/bufr-query#53)) 2. Enable IASI tests - add tests in CMakeList.txt - add test references (testoutput) in obsForge-0.0.0.tgz on EMCRZDM - add test configuration YAMLs (testinput) - mapping file and python configuration YAMLs are added in [SPOC PR#25](NOAA-EMC/spoc#25) --------- Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds setup entry for hera yaml, rocoto script, job script, exglobal script, and task class with stub methods to process marine bufr files.
Tested on hera to successfully run task, which does nothing but print a log.