Add GNSSRO bufr2ioda and json files by XuanliLi-NOAA · Pull Request #1448 · NOAA-EMC/GDASApp

XuanliLi-NOAA · 2025-01-16T19:36:41Z

Description

Add JSON and bufr2ioda python scripts for GNSSRO data. Each GNSSRO mission has a separate ioda obs file.

Companion PRs

N/A

Issues

Refs #750

Automated CI tests to run in Global Workflow

CoryMartin-NOAA · 2025-01-17T15:29:57Z

Should these go in here or in spoc? @emilyhcliu what are your thoughts?

emilyhcliu · 2025-01-17T15:49:20Z

@CoryMartin-NOAA
The json and Python scripts in this PR are written in bufr-query low-level Python API.
These are good to use with the current setup in GDASApp. So, they should be checked in here in GDASApp so Xuanli can keep track of changes and repeat her tests.

In the meanwhile, @nicholasesposito will create these low-level Pythons into one Python script with high-level API and mapping files. These will be checked into SPOC.

Once we have most of the Python Script, mapping files in SPOC, we will work on GDASApp to use them from SPOC.

CoryMartin-NOAA · 2025-01-17T15:50:26Z

ok sounds good thanks @emilyhcliu I just wanted to double check

CoryMartin-NOAA

a few minor comments, and I didn't closely look at the converters but assume they have been thoroughly tested

emilyhcliu · 2025-01-17T16:00:04Z

@XuanliLi-NOAA one question.
Is it possible to write one Python to convert all platforms?
GPSRO data is in one subset. It should be possible to write one Python to handle all platforms.
Are these platforms using the same BUFR table?

No worries, what you have in this PR is OK. But. we should try to create one Python script with high-level Python API for GPSRO.
@nicholasesposito will work with you on this.

Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>

…C/GDASApp into feature/gnssro_bufr2ioda

XuanliLi-NOAA · 2025-01-17T16:28:18Z

@emilyhcliu , it would be great to have one Python script to convert all platforms. Yes, let's do that with the with high-level Python API.

emilyhcliu · 2025-01-17T16:29:56Z

@emilyhcliu , it would be great to have one Python script to convert all platforms. Yes, let's do that with the with high-level Python API.

Do all GPSRO platform use the same bufr table?

XuanliLi-NOAA · 2025-01-17T16:46:15Z

@emilyhcliu, yes, for the test date 00z 2024-02-19, they use the same bufr table. But after June 2024, we have new EUMETSAT Spire data, this has an additional BUFR mnemonic "SASBID". Should we consider that in the high level python API?

@XuanliLi-NOAA Yes, we should include the new one. It is good that all platforms share the same bufr table mnemonics. We can have one mapping file to handle all of them. So, the one Python script with high-level API should be doable for GPSRO.
Thanks for the information.

@emilyhcliu Sounds good, thank you for confirming that.

nicholasesposito

Recommend changing the global attributes, but @emilyhcliu should comment on that. The rest looked fine with the exception of I think changing the data_format description.

XuanliLi-NOAA · 2025-01-24T19:35:59Z

@emilyhcliu and @nicholasesposito data_provider has been corrected in the json files to reflect the actual originating center. data_format has been changed to dataOriginalFormatSpec in the python scripts. I'm not sure if we should replace subsets with platforms, as subsets is a bufr message for the name of the subset.

nicholasesposito

I think just clean up some of the comments unless they are needed. Otherwise no glaring issues from me. Looks very complicated so thank you for taking the time to do all this!

XuanliLi-NOAA · 2025-01-31T23:31:54Z

@nicholasesposito @emilyhcliu Thank you so much, Nick. I cleaned up the python scripts and also uncommented mefr1 to maintain consistency with the other variables.

nicholasesposito

Approved.

* develop: added original date (#1570) Update OCEAN_BASIN_FILE_ORION path for marine ctest (#1569) Fixing window_begin and end in yaml for redating ADT (#1565) Save LETKF mean analysis and mean increment for verification (#1566) Update JEDI hashes (20250324) (#1567) Removes obsolete json files from marine bufr2ioda (#1564) Add hybrid 4denvar as a phase3 validation configuration (#1552) Bugfix/ocn ecen (#1550) Bugfix SSS obs for LETKF (#1560) Bugfix ocean ensemble recentering (#1558) apply ci path patch only when running g-w ctests (#1554) Update JEDI hashes (20250317) (#1531) add option to create_experiment.py in create_exp.sh (#1546) Update jcb hashes and support initial IODA stats capability (#1547) Fix GW-CI CTest dependencies (#1534) Converts prepoceanobs to use JCB-style obs list and config files (#1527) JEDI-based ensemble recentering and analysis calculation (#1488) tools for phase3 validation (#1526) Update JEDI hashes (20250306) (#1520) New jcb-gdas # and insitu obs prep bug fix (#1523) Marine in situ obs update (SSS, moorings, drifters) (#1511) Save marine background and analysis spread (#1517) Minor build and install updates and ctest cleanup (#1512) correct logic for upper loop limit in chem_diagb (#1516) Incorporate multi-cycle concatination and separation of variables in post-BUFR observations (#1504) Update JEDI hashes (20250225) (#1475) Add a depth dependent e-fold scale (#1494) Update jcb and da-utils hashes to support land DA. (#1509) Add options to inflate the ensemble while recentering (#1501) Add components to use jcb yamls for marine letkf job (#1486) Add GNSSRO bufr2ioda and json files (#1448) Removal of the marine verification tools (#1499) update logic to identify MSU machine (#1498) IODA Converter with In-Situ Observations Concatenation and ObsError Inflation (#1472) Add sea ice recentering (#1478) marine vrfy bugfix (#1483) add LD_LIBRARY_PATH patch for wcoss2 build (#1484) Bugfix for ensemble recenter + add a test (#1477) Add code changes for the implementation of the low-resolution B (#1441) update spack-stack path in Hera modulefiles (#1471) fixed glider output filename (#1469) add gdas_waveinit to C48mx500_3DVarAOWCDA ctest suite (#1470) update jcb-gdas hash 20250128 (#1466) Time series of omb stats and html doc (#1463) Update marine vrfy HTML codes (#1464) Automate the verify tools to submit from start pdy to end pdy (#1458) Update an offline marine vrfy tool (#1447) Update JEDI hashes (20250127) (#1460)

# Description Add JSON and bufr2ioda python scripts for GNSSRO data. Each GNSSRO mission has a separate ioda obs file. # Companion PRs N/A # Issues Refs #750 # Automated CI tests to run in Global Workflow  - [ ] atm_jjob  - [ ] C96C48_ufs_hybatmDA  - [ ] C96C48_hybatmaerosnowDA  - [ ] C48mx500_3DVarAOWCDA  - [ ] C48mx500_hybAOWCDA  - [ ] C96C48_hybatmDA

Add GNSSRO bufr2ioda and json files

4dca3a9

XuanliLi-NOAA self-assigned this Jan 16, 2025

XuanliLi-NOAA added 3 commits January 16, 2025 14:04

update bufr2ioda scripts

89f0e43

update bufr2ioda python scripts

ba3fb3e

update bufr2ioda scripts

a396f11

XuanliLi-NOAA requested review from ADCollard, CoryMartin-NOAA, emilyhcliu, jianjunj and nicholasesposito January 16, 2025 20:16

CoryMartin-NOAA reviewed Jan 17, 2025

View reviewed changes

Comment thread parm/atm/jcb-prototype_3dvar.yaml.j2 Outdated

Comment thread ush/ioda/bufr2ioda/bufr2ioda_gnssro_cosmic2.py Outdated

XuanliLi-NOAA and others added 3 commits January 17, 2025 11:13

Update parm/atm/jcb-prototype_3dvar.yaml.j2

fd59b1c

Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>

remove the date line

c8e37bf

Merge branch 'feature/gnssro_bufr2ioda' of https://github.com/NOAA-EM…

2024752

…C/GDASApp into feature/gnssro_bufr2ioda

nicholasesposito reviewed Jan 22, 2025

View reviewed changes

Comment thread parm/ioda/bufr2ioda/bufr2ioda_gnssro_cosmic2.json Outdated

Comment thread ush/ioda/bufr2ioda/bufr2ioda_gnssro_cosmic2.py

modified variable names in json and python scripts

9fdbe6a

nicholasesposito previously approved these changes Jan 31, 2025

View reviewed changes

Comment thread ush/ioda/bufr2ioda/bufr2ioda_gnssro_cosmic2.py Outdated

Comment thread ush/ioda/bufr2ioda/bufr2ioda_gnssro_cosmic2.py Outdated

clean up the python script

9f952b0

XuanliLi-NOAA dismissed nicholasesposito’s stale review via 9f952b0 January 31, 2025 23:27

emilyhcliu approved these changes Feb 5, 2025

View reviewed changes

nicholasesposito approved these changes Feb 6, 2025

View reviewed changes

Merge branch 'develop' into feature/gnssro_bufr2ioda

7510a5b

CoryMartin-NOAA requested review from CoryMartin-NOAA, emilyhcliu and nicholasesposito February 19, 2025 20:10

CoryMartin-NOAA approved these changes Feb 19, 2025

View reviewed changes

nicholasesposito approved these changes Feb 19, 2025

View reviewed changes

CoryMartin-NOAA merged commit bec351a into develop Feb 19, 2025

CoryMartin-NOAA deleted the feature/gnssro_bufr2ioda branch February 19, 2025 20:24

Conversation

XuanliLi-NOAA commented Jan 16, 2025

Description

Companion PRs

Issues

Automated CI tests to run in Global Workflow

Uh oh!

CoryMartin-NOAA commented Jan 17, 2025

Uh oh!

emilyhcliu commented Jan 17, 2025

Uh oh!

CoryMartin-NOAA commented Jan 17, 2025

Uh oh!

CoryMartin-NOAA left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

emilyhcliu commented Jan 17, 2025

Uh oh!

XuanliLi-NOAA commented Jan 17, 2025

Uh oh!

emilyhcliu commented Jan 17, 2025

Uh oh!

XuanliLi-NOAA commented Jan 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nicholasesposito left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

XuanliLi-NOAA commented Jan 24, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nicholasesposito left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

XuanliLi-NOAA commented Jan 31, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

nicholasesposito left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

4 participants

XuanliLi-NOAA commented Jan 17, 2025 •

edited

Loading

XuanliLi-NOAA commented Jan 24, 2025 •

edited

Loading

XuanliLi-NOAA commented Jan 31, 2025 •

edited

Loading