Added seaice and sea surface salinity database by apchoiCMD · Pull Request #57 · NOAA-EMC/obsForge

apchoiCMD · 2025-04-23T16:12:31Z

This PR adds an additional database to obsForge for seaice, sss of smap and smos.

Resolves #55
Partially #56

guillaumevernieres · 2025-04-23T17:00:43Z

Code norms are failing @apchoiCMD

guillaumevernieres

I'm pretty sure you don't need to modify the base class to make the new databases work.

guillaumevernieres

A few minor issues. Check the comments.
For the provider's name, RADS and GHRSST were quite straight forward to pick, I understand that may not be the case for sss and ice, but we need something better than the satellite's name.

Copilot

Pull Request Overview

This PR adds a new database and related tests for sea ice and sea surface salinity observations from SMAP, SMOS, and NESDIS AMSR2 and integrates them into the obsForge workflow.

Introduces new test modules for SMOS, SMAP, and NESDIS AMSR2 databases.
Adds a new database class for NESDIS AMSR2 and updates provider configuration and task processing to include it.
Updates configuration files to reflect new provider settings and environment changes.

Reviewed Changes

Copilot reviewed 12 out of 12 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_smos_database.py	New tests for SMOS database ingestion and filename parsing.
tests/test_smap_database.py	New tests for SMAP database ingestion and filename parsing.
tests/test_nesdis_amsr2_database.py	New tests for NESDIS AMSR2 database ingestion and filename parsing.
task/providers.py	Adds provider support for nesdis_amsr2 with a new debug log statement.
task/marine_prepobs.py	Initializes and ingests files for nesdis_amsr2 and updates obs-space processing.
obsdb/smos_db.py, obsdb/smap_db.py	Database classes for SMOS and SMAP with filename parsing and ingestion.
obsdb/nesdis_amsr2_db.py	New database class for NESDIS AMSR2 with its own filename parsing logic.
parm/config.yaml, parm/config.orion.yaml	Updates to configuration reflecting new paths and provider settings.
parm/config.hera.yaml	New configuration file with updated provider entries.

guillaumevernieres

See comments

guillaumevernieres

see copilot's review comment

…sts (#12) 1. Simplified ctest structure. My initial version is very stupid and inefficient! e.g. initial ctest list for test_obsforge_satobs_satwnd_amv_seviri looks like the following: ``` Test #38: test_obsforge_satobs_satwnd_amv_seviri_m8_bufr2netcdf Test #39: test_obsforge_satobs_satwnd_amv_seviri_m8_bufr2netcdf_mpi4 Test #40: test_obsforge_satobs_satwnd_amv_seviri_m8_script2netcdf Test #41: test_obsforge_satobs_satwnd_amv_seviri_m8_script2netcdf_mpi4 Test #42: test_obsforge_satobs_satwnd_amv_seviri_m8_bufr4backend Test #43: test_obsforge_satobs_satwnd_amv_seviri_m8_bufr4backend_mpi4 Test #44: test_obsforge_satobs_satwnd_amv_seviri_m8_script4backend Test #45: test_obsforge_satobs_satwnd_amv_seviri_m8_script4backend_mpi4 Test #46: test_obsforge_satobs_satwnd_amv_seviri_m9_bufr2netcdf Test #47: test_obsforge_satobs_satwnd_amv_seviri_m9_bufr2netcdf_mpi4 Test #48: test_obsforge_satobs_satwnd_amv_seviri_m9_script2netcdf Test #49: test_obsforge_satobs_satwnd_amv_seviri_m9_script2netcdf_mpi4 Test #50: test_obsforge_satobs_satwnd_amv_seviri_m9_bufr4backend Test #51: test_obsforge_satobs_satwnd_amv_seviri_m9_bufr4backend_mpi4 Test #52: test_obsforge_satobs_satwnd_amv_seviri_m9_script4backend Test #53: test_obsforge_satobs_satwnd_amv_seviri_m9_script4backend_mpi4 Test #54: test_obsforge_satobs_satwnd_amv_seviri_m10_bufr2netcdf Test #55: test_obsforge_satobs_satwnd_amv_seviri_m10_bufr2netcdf_mpi4 Test #56: test_obsforge_satobs_satwnd_amv_seviri_m10_script2netcdf Test #57: test_obsforge_satobs_satwnd_amv_seviri_m10_script2netcdf_mpi4 Test #58: test_obsforge_satobs_satwnd_amv_seviri_m10_bufr4backend Test #59: test_obsforge_satobs_satwnd_amv_seviri_m10_bufr4backend_mpi4 Test #60: test_obsforge_satobs_satwnd_amv_seviri_m10_script4backend Test #61: test_obsforge_satobs_satwnd_amv_seviri_m10_script4backend_mpi4 Test #62: test_obsforge_satobs_satwnd_amv_seviri_m11_bufr2netcdf Test #63: test_obsforge_satobs_satwnd_amv_seviri_m11_bufr2netcdf_mpi4 Test #64: test_obsforge_satobs_satwnd_amv_seviri_m11_script2netcdf Test #65: test_obsforge_satobs_satwnd_amv_seviri_m11_script2netcdf_mpi4 Test #66: test_obsforge_satobs_satwnd_amv_seviri_m11_bufr4backend Test #67: test_obsforge_satobs_satwnd_amv_seviri_m11_bufr4backend_mpi4 Test #68: test_obsforge_satobs_satwnd_amv_seviri_m11_script4backend Test #69: test_obsforge_satobs_satwnd_amv_seviri_m11_script4backend_mpi4 ``` The simplified one looks like the following for satwnd_amv_seviri: ``` Start 22: test_obsforge_satobs_satwnd_amv_seviri_bufr2netcdf 7/38 Test #22: test_obsforge_satobs_satwnd_amv_seviri_bufr2netcdf ........... Passed 16.69 sec Start 23: test_obsforge_satobs_satwnd_amv_seviri_bufr2netcdf_mpi4 8/38 Test #23: test_obsforge_satobs_satwnd_amv_seviri_bufr2netcdf_mpi4 ...... Passed 9.47 sec Start 24: test_obsforge_satobs_satwnd_amv_seviri_script2netcdf 9/38 Test #24: test_obsforge_satobs_satwnd_amv_seviri_script2netcdf ......... Passed 18.98 sec Start 25: test_obsforge_satobs_satwnd_amv_seviri_script2netcdf_mpi4 10/38 Test #25: test_obsforge_satobs_satwnd_amv_seviri_script2netcdf_mpi4 .... Passed 13.31 sec Start 26: test_obsforge_satobs_satwnd_amv_seviri_bufr4backend 11/38 Test #26: test_obsforge_satobs_satwnd_amv_seviri_bufr4backend .......... Passed 18.95 sec Start 27: test_obsforge_satobs_satwnd_amv_seviri_bufr4backend_mpi4 12/38 Test #27: test_obsforge_satobs_satwnd_amv_seviri_bufr4backend_mpi4 ..... Passed 11.18 sec Start 28: test_obsforge_satobs_satwnd_amv_seviri_script4backend 13/38 Test #28: test_obsforge_satobs_satwnd_amv_seviri_script4backend ........ Passed 22.38 sec Start 29: test_obsforge_satobs_satwnd_amv_seviri_script4backend_mpi4 14/38 Test #29: test_obsforge_satobs_satwnd_amv_seviri_script4backend_mpi4 ... Passed 16.85 sec ``` The test structure is simplified in that the test for each data type, and each configuration only needs to be tested once because the `bufr_comp.sh` has been modified to handle comparisons of output files from various satellite platforms for the same sensor type. (See [bufr-query PR #53](NOAA-EMC/bufr-query#53)) 2. Enable IASI tests - add tests in CMakeList.txt - add test references (testoutput) in obsForge-0.0.0.tgz on EMCRZDM - add test configuration YAMLs (testinput) - mapping file and python configuration YAMLs are added in [SPOC PR#25](NOAA-EMC/spoc#25) --------- Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>

apchoiCMD and others added 7 commits April 15, 2025 13:31

initial

abd788a

initial config

ce7a139

dcom folder

00aa670

add the sss database

2dd51a6

initial processing test

c44112a

2nd checking

dc4393a

add the database

7fbfb18

apchoiCMD requested a review from guillaumevernieres April 23, 2025 16:12

guillaumevernieres requested changes Apr 23, 2025

View reviewed changes

Comment thread ush/python/pyobsforge/obsdb/amsr2_db.py Outdated

Comment thread ush/python/pyobsforge/obsdb/amsr2_db.py

Comment thread ush/python/pyobsforge/obsdb/obsdb.py Outdated

guillaumevernieres mentioned this pull request Apr 23, 2025

Use logging instead of print #58

Closed

apchoiCMD added 8 commits April 23, 2025 17:36

py-norm_1

a0628e6

py-test_2

0d6cc7d

py-norm_3

b7b215b

py-norm_4

547497f

py-norm_5

8af6dfa

py-norm_6

ae9a031

update globing amsr2

dbae9f2

remove variable

d0373ba

apchoiCMD marked this pull request as ready for review April 23, 2025 18:40

apchoiCMD requested a review from guillaumevernieres April 23, 2025 18:40

guillaumevernieres requested changes Apr 23, 2025

View reviewed changes

Comment thread ush/python/pyobsforge/obsdb/obsdb.py Outdated

Comment thread ush/python/pyobsforge/task/marine_prepobs.py Outdated

apchoiCMD added 6 commits April 23, 2025 19:52

address comments

ba96dfb

correct provider name

281a351

add name -nesdis- to seaice product

ca36d10

revise config

383f63a

revise kwargs

6cb2318

config, provider and obs_space

4929d94

apchoiCMD changed the title ~~Added seaice and salinity database~~ Added seaice and sea surface salinity database Apr 24, 2025

clean up and obs_type convention

da77a0c

apchoiCMD added 2 commits April 28, 2025 16:15

obs_space correction

b16a4ed

fix pytest

223851d

apchoiCMD requested a review from guillaumevernieres April 28, 2025 16:26

marine icec kwargs

bc63ab1

guillaumevernieres requested a review from Copilot April 28, 2025 18:47

Copilot AI reviewed Apr 28, 2025

View reviewed changes

Comment thread ush/python/pyobsforge/task/providers.py Outdated

Comment thread ush/python/pyobsforge/obsdb/nesdis_amsr2_db.py Outdated

guillaumevernieres reviewed Apr 28, 2025

View reviewed changes

apchoiCMD added 5 commits April 29, 2025 15:44

address comments

a2cb073

clean up

4c10563

config yaml for hercules

c6a117b

smap success

6050ede

pytest

c8ceb9f

guillaumevernieres reviewed Apr 29, 2025

View reviewed changes

apchoiCMD added 2 commits April 30, 2025 14:56

fix pytest

539a129

address copilot's review

7f7c7fd

guillaumevernieres approved these changes Apr 30, 2025

View reviewed changes

guillaumevernieres merged commit 5f8206e into main Apr 30, 2025
2 checks passed

guillaumevernieres deleted the feature/all-databse branch April 30, 2025 15:49

Conversation

apchoiCMD commented Apr 23, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

guillaumevernieres commented Apr 23, 2025

Uh oh!

guillaumevernieres left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

guillaumevernieres left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Reviewed Changes

Uh oh!

Uh oh!

Uh oh!

guillaumevernieres left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

guillaumevernieres left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

apchoiCMD commented Apr 23, 2025 •

edited

Loading