Implement the AOD task#48
Merged
Merged
Conversation
Contributor
guillaumevernieres
commented
Apr 2, 2025
guillaumevernieres
commented
- fixes Make the AOD job do something #47
Contributor
There was a problem hiding this comment.
Pull Request Overview
This PR implements the AOD task and fixes issue #47 by introducing a new module to run the nc2ioda conversion while refactoring related tasks and updating configuration and tests.
- Added the run_nc2ioda.py module to encapsulate the nc2ioda conversion logic.
- Refactored both marine and aerosol preparation tasks to use the new run_nc2ioda function.
- Updated the database file handling, test scripts, and configuration to support the AOD task.
Reviewed Changes
Copilot reviewed 7 out of 8 changed files in this pull request and generated no comments.
Show a summary per file
| File | Description |
|---|---|
| ush/python/pyobsforge/task/run_nc2ioda.py | New module to execute the nc2ioda conversion with improved error logging. |
| ush/python/pyobsforge/task/marine_prepobs.py | Removed duplicated code and invoked run_nc2ioda for processing valid files. |
| ush/python/pyobsforge/task/aero_prepobs.py | Updated to initialize and invoke run_nc2ioda and adjusted variable naming. |
| ush/python/pyobsforge/obsdb/obsdb.py | Modified get_valid_files method to copy files to a destination directory. |
| scripts/tests/test_exobsforge_global_dump.py | Updated test scripts to run both marine and AOD dump scripts with clearer logging. |
| scripts/exobsforge_global_aod_dump.py | Adjusted instance variable naming when invoking the aerosol observation preparation. |
| parm/config.yaml | Added configuration entries for the AOD dump task. |
Files not reviewed (1)
- parm/nc2ioda/nc2ioda.yaml.j2: Language not supported
Comments suppressed due to low confidence (3)
ush/python/pyobsforge/task/run_nc2ioda.py:40
- [nitpick] The multiline f-string for logging the exception may not format as expected. Consider rewriting it as a single-line f-string, e.g. logger.warning(f"ioda converter failed with error {e}, return code {e.returncode}") for clarity.
logger.warning(f"ioda converter failed with error {e}, \
ush/python/pyobsforge/task/aero_prepobs.py:30
- [nitpick] The instance variable name 'aeroObs' does not follow the standard snake_case convention. Consider renaming it to 'aerosol_obs' for improved readability and consistency with Python naming standards.
aeroObs = AerosolObsPrep(config)
scripts/tests/test_exobsforge_global_dump.py:71
- [nitpick] Avoid using 'exec' as a variable name since it shadows the built-in Python function. Consider renaming it to something like 'script_path'.
exec = Path(__file__).parent.parent / script_name
Contributor
|
There is a bug to read Level 2 product something like below; |
apchoiCMD
approved these changes
Apr 3, 2025
Contributor
apchoiCMD
left a comment
apchoiCMD
left a comment
There was a problem hiding this comment.
L3 Product only works!
CoryMartin-NOAA
added a commit
that referenced
this pull request
Oct 21, 2025
…sts (#12) 1. Simplified ctest structure. My initial version is very stupid and inefficient! e.g. initial ctest list for test_obsforge_satobs_satwnd_amv_seviri looks like the following: ``` Test #38: test_obsforge_satobs_satwnd_amv_seviri_m8_bufr2netcdf Test #39: test_obsforge_satobs_satwnd_amv_seviri_m8_bufr2netcdf_mpi4 Test #40: test_obsforge_satobs_satwnd_amv_seviri_m8_script2netcdf Test #41: test_obsforge_satobs_satwnd_amv_seviri_m8_script2netcdf_mpi4 Test #42: test_obsforge_satobs_satwnd_amv_seviri_m8_bufr4backend Test #43: test_obsforge_satobs_satwnd_amv_seviri_m8_bufr4backend_mpi4 Test #44: test_obsforge_satobs_satwnd_amv_seviri_m8_script4backend Test #45: test_obsforge_satobs_satwnd_amv_seviri_m8_script4backend_mpi4 Test #46: test_obsforge_satobs_satwnd_amv_seviri_m9_bufr2netcdf Test #47: test_obsforge_satobs_satwnd_amv_seviri_m9_bufr2netcdf_mpi4 Test #48: test_obsforge_satobs_satwnd_amv_seviri_m9_script2netcdf Test #49: test_obsforge_satobs_satwnd_amv_seviri_m9_script2netcdf_mpi4 Test #50: test_obsforge_satobs_satwnd_amv_seviri_m9_bufr4backend Test #51: test_obsforge_satobs_satwnd_amv_seviri_m9_bufr4backend_mpi4 Test #52: test_obsforge_satobs_satwnd_amv_seviri_m9_script4backend Test #53: test_obsforge_satobs_satwnd_amv_seviri_m9_script4backend_mpi4 Test #54: test_obsforge_satobs_satwnd_amv_seviri_m10_bufr2netcdf Test #55: test_obsforge_satobs_satwnd_amv_seviri_m10_bufr2netcdf_mpi4 Test #56: test_obsforge_satobs_satwnd_amv_seviri_m10_script2netcdf Test #57: test_obsforge_satobs_satwnd_amv_seviri_m10_script2netcdf_mpi4 Test #58: test_obsforge_satobs_satwnd_amv_seviri_m10_bufr4backend Test #59: test_obsforge_satobs_satwnd_amv_seviri_m10_bufr4backend_mpi4 Test #60: test_obsforge_satobs_satwnd_amv_seviri_m10_script4backend Test #61: test_obsforge_satobs_satwnd_amv_seviri_m10_script4backend_mpi4 Test #62: test_obsforge_satobs_satwnd_amv_seviri_m11_bufr2netcdf Test #63: test_obsforge_satobs_satwnd_amv_seviri_m11_bufr2netcdf_mpi4 Test #64: test_obsforge_satobs_satwnd_amv_seviri_m11_script2netcdf Test #65: test_obsforge_satobs_satwnd_amv_seviri_m11_script2netcdf_mpi4 Test #66: test_obsforge_satobs_satwnd_amv_seviri_m11_bufr4backend Test #67: test_obsforge_satobs_satwnd_amv_seviri_m11_bufr4backend_mpi4 Test #68: test_obsforge_satobs_satwnd_amv_seviri_m11_script4backend Test #69: test_obsforge_satobs_satwnd_amv_seviri_m11_script4backend_mpi4 ``` The simplified one looks like the following for satwnd_amv_seviri: ``` Start 22: test_obsforge_satobs_satwnd_amv_seviri_bufr2netcdf 7/38 Test #22: test_obsforge_satobs_satwnd_amv_seviri_bufr2netcdf ........... Passed 16.69 sec Start 23: test_obsforge_satobs_satwnd_amv_seviri_bufr2netcdf_mpi4 8/38 Test #23: test_obsforge_satobs_satwnd_amv_seviri_bufr2netcdf_mpi4 ...... Passed 9.47 sec Start 24: test_obsforge_satobs_satwnd_amv_seviri_script2netcdf 9/38 Test #24: test_obsforge_satobs_satwnd_amv_seviri_script2netcdf ......... Passed 18.98 sec Start 25: test_obsforge_satobs_satwnd_amv_seviri_script2netcdf_mpi4 10/38 Test #25: test_obsforge_satobs_satwnd_amv_seviri_script2netcdf_mpi4 .... Passed 13.31 sec Start 26: test_obsforge_satobs_satwnd_amv_seviri_bufr4backend 11/38 Test #26: test_obsforge_satobs_satwnd_amv_seviri_bufr4backend .......... Passed 18.95 sec Start 27: test_obsforge_satobs_satwnd_amv_seviri_bufr4backend_mpi4 12/38 Test #27: test_obsforge_satobs_satwnd_amv_seviri_bufr4backend_mpi4 ..... Passed 11.18 sec Start 28: test_obsforge_satobs_satwnd_amv_seviri_script4backend 13/38 Test #28: test_obsforge_satobs_satwnd_amv_seviri_script4backend ........ Passed 22.38 sec Start 29: test_obsforge_satobs_satwnd_amv_seviri_script4backend_mpi4 14/38 Test #29: test_obsforge_satobs_satwnd_amv_seviri_script4backend_mpi4 ... Passed 16.85 sec ``` The test structure is simplified in that the test for each data type, and each configuration only needs to be tested once because the `bufr_comp.sh` has been modified to handle comparisons of output files from various satellite platforms for the same sensor type. (See [bufr-query PR #53](NOAA-EMC/bufr-query#53)) 2. Enable IASI tests - add tests in CMakeList.txt - add test references (testoutput) in obsForge-0.0.0.tgz on EMCRZDM - add test configuration YAMLs (testinput) - mapping file and python configuration YAMLs are added in [SPOC PR#25](NOAA-EMC/spoc#25) --------- Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.