Incorporate multi-cycle concatination and separation of variables in post-BUFR observations by AndrewEichmann-NOAA · Pull Request #1504 · NOAA-EMC/GDASApp

AndrewEichmann-NOAA · 2025-02-20T18:33:37Z

Description

Implements modifications to concatinate IODA observation files converted from BUFR format from multiple cycles in a window to a single file for the current cycle
Separates temperature from salinity observations in Argo floats so that they are delivered to obs under COMROOT as two separate IODA files

Tested with C48mx500_3DVarAOWCDA and C48mx500_hybAOWCDA on Hera with NOAA-EMC/jcb-gdas#78 and obs spaces enabled in OBS_LIST. Aside from passing test, files gdas.t00z.insitu_salt_profile_argo.2021032500.nc4 and gdas.t00z.insitu_temp_profile_argo.2021032500.nc4 are delivered to gdas.20210325/00/obs bearing plausible-looking values for salt and temperature respectively, and seem to be assimilated. No quantitative analysis has been performed on these obs.

Companion PRs

Not necessary for merging and passing ctests, but if new obs spaces get included the following is needed:
NOAA-EMC/jcb-gdas#78

Issues

Mostly resolves #1132

Automated CI tests to run in Global Workflow

C48mx500_3DVarAOWCDA
C48mx500_hybAOWCDA

guillaumevernieres

The obs spaces for insitu temp and salt are not setup correctly. I added links to simple examples from the soca repo for guidance.

…DASApp into feature/insitu-concat

emcbot · 2025-02-21T23:47:08Z

Automated GW-GDASApp Testing Results:
Machine: hera

Start: Fri Feb 21 20:12:59 UTC 2025 on hfe01
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Fri Feb 21 20:58:03 UTC 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp -E atm_jjob|C96C48_ufs_hybatmDA|C96C48_hybatmaerosnowDA|C96C48_hybatmDA
Tests:                                  *Failed*
Tests: Failed at Fri Feb 21 23:47:07 UTC 2025
Tests: 84% tests passed, 9 tests failed out of 56
	2044 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlinit_202103250000 (Failed)
	2045 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlvar_202103250000 (Failed)
	2046 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlchkpt_202103250000 (Failed)
	2047 - test_gdasapp_C48mx500_3DVarAOWCDA_gdas_marineanlfinal_202103250000 (Failed)
	2055 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlletkf_202103250000 (Failed)
	2056 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlinit_202103250000 (Failed)
	2057 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlvar_202103250000 (Failed)
	2059 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlchkpt_202103250000 (Failed)
	2060 - test_gdasapp_C48mx500_hybAOWCDA_gdas_marineanlfinal_202103250000 (Failed)
Tests: see output at /scratch1/NCEPDEV/da/role.jedipara/CI/GDASApp/workflow/PR/1504/global-workflow/sorc/gdas.cd/build/log.ctest

AndrewEichmann-NOAA · 2025-02-24T15:16:47Z

The last CI seems to have been started before the latest commits (particularly to obs_list)

emcbot · 2025-02-24T17:04:29Z

Automated GW-GDASApp Testing Results:
Machine: hera

Start: Mon Feb 24 15:45:02 UTC 2025 on hfe10
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Mon Feb 24 16:29:13 UTC 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp -E atm_jjob|C96C48_ufs_hybatmDA|C96C48_hybatmaerosnowDA|C96C48_hybatmDA
Tests:                                 *SUCCESS*
Tests: Completed at Mon Feb 24 17:04:28 UTC 2025
Tests: 100% tests passed, 0 tests failed out of 56

apchoiCMD

look good-

AndrewEichmann-NOAA · 2025-02-26T18:43:31Z

Does this need any more changes?

apchoiCMD · 2025-02-26T18:51:47Z

Does this need any more changes?

Let me quickly check one more time-

guillaumevernieres

Just 1 minor thing @AndrewEichmann-NOAA

…DASApp into feature/insitu-concat

guillaumevernieres

Thanks @AndrewEichmann-NOAA

…post-BUFR observations (#1504) # Description - Implements modifications to concatinate IODA observation files converted from BUFR format from multiple cycles in a window to a single file for the current cycle - Separates temperature from salinity observations in Argo floats so that they are delivered to obs under COMROOT as two separate IODA files Tested with `C48mx500_3DVarAOWCDA` and `C48mx500_hybAOWCDA` on Hera with NOAA-EMC/jcb-gdas#78 and obs spaces enabled in `OBS_LIST`. Aside from passing test, files `gdas.t00z.insitu_salt_profile_argo.2021032500.nc4` and `gdas.t00z.insitu_temp_profile_argo.2021032500.nc4` are delivered to `gdas.20210325/00/obs` bearing plausible-looking values for salt and temperature respectively, and seem to be assimilated. No quantitative analysis has been performed on these obs. # Companion PRs Not necessary for merging and passing ctests, but if new obs spaces get included the following is needed: NOAA-EMC/jcb-gdas#78 # Issues Mostly resolves #1132 # Automated CI tests to run in Global Workflow - [x] C48mx500_3DVarAOWCDA  - [x] C48mx500_hybAOWCDA  --------- Co-authored-by: Guillaume Vernieres <guillaume.vernieres@noaa.gov>

* develop: added original date (#1570) Update OCEAN_BASIN_FILE_ORION path for marine ctest (#1569) Fixing window_begin and end in yaml for redating ADT (#1565) Save LETKF mean analysis and mean increment for verification (#1566) Update JEDI hashes (20250324) (#1567) Removes obsolete json files from marine bufr2ioda (#1564) Add hybrid 4denvar as a phase3 validation configuration (#1552) Bugfix/ocn ecen (#1550) Bugfix SSS obs for LETKF (#1560) Bugfix ocean ensemble recentering (#1558) apply ci path patch only when running g-w ctests (#1554) Update JEDI hashes (20250317) (#1531) add option to create_experiment.py in create_exp.sh (#1546) Update jcb hashes and support initial IODA stats capability (#1547) Fix GW-CI CTest dependencies (#1534) Converts prepoceanobs to use JCB-style obs list and config files (#1527) JEDI-based ensemble recentering and analysis calculation (#1488) tools for phase3 validation (#1526) Update JEDI hashes (20250306) (#1520) New jcb-gdas # and insitu obs prep bug fix (#1523) Marine in situ obs update (SSS, moorings, drifters) (#1511) Save marine background and analysis spread (#1517) Minor build and install updates and ctest cleanup (#1512) correct logic for upper loop limit in chem_diagb (#1516) Incorporate multi-cycle concatination and separation of variables in post-BUFR observations (#1504) Update JEDI hashes (20250225) (#1475) Add a depth dependent e-fold scale (#1494) Update jcb and da-utils hashes to support land DA. (#1509) Add options to inflate the ensemble while recentering (#1501) Add components to use jcb yamls for marine letkf job (#1486) Add GNSSRO bufr2ioda and json files (#1448) Removal of the marine verification tools (#1499) update logic to identify MSU machine (#1498) IODA Converter with In-Situ Observations Concatenation and ObsError Inflation (#1472) Add sea ice recentering (#1478) marine vrfy bugfix (#1483) add LD_LIBRARY_PATH patch for wcoss2 build (#1484) Bugfix for ensemble recenter + add a test (#1477) Add code changes for the implementation of the low-resolution B (#1441) update spack-stack path in Hera modulefiles (#1471) fixed glider output filename (#1469) add gdas_waveinit to C48mx500_3DVarAOWCDA ctest suite (#1470) update jcb-gdas hash 20250128 (#1466) Time series of omb stats and html doc (#1463) Update marine vrfy HTML codes (#1464) Automate the verify tools to submit from start pdy to end pdy (#1458) Update an offline marine vrfy tool (#1447) Update JEDI hashes (20250127) (#1460)

…post-BUFR observations (#1504) # Description - Implements modifications to concatinate IODA observation files converted from BUFR format from multiple cycles in a window to a single file for the current cycle - Separates temperature from salinity observations in Argo floats so that they are delivered to obs under COMROOT as two separate IODA files Tested with `C48mx500_3DVarAOWCDA` and `C48mx500_hybAOWCDA` on Hera with NOAA-EMC/jcb-gdas#78 and obs spaces enabled in `OBS_LIST`. Aside from passing test, files `gdas.t00z.insitu_salt_profile_argo.2021032500.nc4` and `gdas.t00z.insitu_temp_profile_argo.2021032500.nc4` are delivered to `gdas.20210325/00/obs` bearing plausible-looking values for salt and temperature respectively, and seem to be assimilated. No quantitative analysis has been performed on these obs. # Companion PRs Not necessary for merging and passing ctests, but if new obs spaces get included the following is needed: NOAA-EMC/jcb-gdas#78 # Issues Mostly resolves #1132 # Automated CI tests to run in Global Workflow - [x] C48mx500_3DVarAOWCDA  - [x] C48mx500_hybAOWCDA  --------- Co-authored-by: Guillaume Vernieres <guillaume.vernieres@noaa.gov>

AndrewEichmann-NOAA added 3 commits February 20, 2025 16:18

like athena from zeus's forehead

5b93376

Merge branch 'develop' into feature/insitu-concat

18054ef

remove comments

4c987cb

AndrewEichmann-NOAA requested a review from guillaumevernieres February 20, 2025 18:33

AndrewEichmann-NOAA added 2 commits February 20, 2025 18:37

style points

45416ea

style points

92e8e6d

guillaumevernieres requested changes Feb 21, 2025

View reviewed changes

Comment thread parm/soca/obs/config/insitu_salt_profile_argo.yaml

Comment thread parm/soca/obs/config/insitu_temp_profile_argo.yaml Outdated

Comment thread parm/soca/obs/obs_list.yaml Outdated

AndrewEichmann-NOAA and others added 2 commits February 21, 2025 18:59

corrections to configs, filename aligment

6f5dbf7

Merge branch 'develop' into feature/insitu-concat

055a80f

guillaumevernieres assigned AndrewEichmann-NOAA Feb 21, 2025

guillaumevernieres added the hera-GW-RT Queue for automated testing with global-workflow on Hera label Feb 21, 2025

emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Feb 21, 2025

AndrewEichmann-NOAA added 3 commits February 21, 2025 22:01

coordination of input and output filenames

3876928

Merge branch 'feature/insitu-concat' of https://github.com/NOAA-EMC/G…

2efb0c3

…DASApp into feature/insitu-concat

Merge branch 'develop' into feature/insitu-concat

89a056e

AndrewEichmann-NOAA requested a review from guillaumevernieres February 21, 2025 22:09

emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Feb 21, 2025

AndrewEichmann-NOAA added hera-GW-RT Queue for automated testing with global-workflow on Hera and removed hera-GW-RT-Failed Automated testing with global-workflow failed on Hera labels Feb 24, 2025

emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera and removed hera-GW-RT Queue for automated testing with global-workflow on Hera labels Feb 24, 2025

emcbot added hera-GW-RT-Passed Automated testing with global-workflow successful on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Feb 24, 2025

guillaumevernieres previously approved these changes Feb 24, 2025

View reviewed changes

guillaumevernieres requested a review from JohnSteffen-NOAA February 24, 2025 17:55

guillaumevernieres requested a review from apchoiCMD February 24, 2025 17:55

apchoiCMD previously approved these changes Feb 24, 2025

View reviewed changes

JohnSteffen-NOAA reviewed Feb 24, 2025

View reviewed changes

Comment thread parm/soca/obs/config/insitu_salt_profile_argo.yaml Outdated

JohnSteffen-NOAA previously approved these changes Feb 24, 2025

View reviewed changes

deleting commented-out bits

8975174

AndrewEichmann-NOAA dismissed stale reviews from JohnSteffen-NOAA, apchoiCMD, and guillaumevernieres via 8975174 February 25, 2025 15:49

AndrewEichmann-NOAA and others added 2 commits February 25, 2025 15:52

removed comments

fe2669d

Merge branch 'develop' into feature/insitu-concat

a26fb49

AndrewEichmann-NOAA requested review from JohnSteffen-NOAA and guillaumevernieres February 25, 2025 15:54

apchoiCMD self-requested a review February 25, 2025 16:25

apchoiCMD previously approved these changes Feb 25, 2025

View reviewed changes

JohnSteffen-NOAA previously approved these changes Feb 26, 2025

View reviewed changes

Merge branch 'develop' into feature/insitu-concat

2ddd8bf

guillaumevernieres reviewed Feb 26, 2025

View reviewed changes

Comment thread parm/soca/obsprep/obsprep_config.yaml Outdated

AndrewEichmann-NOAA added 2 commits February 26, 2025 20:50

remove old argo obs space

e628903

Merge branch 'feature/insitu-concat' of https://github.com/NOAA-EMC/G…

1cf85fb

…DASApp into feature/insitu-concat

AndrewEichmann-NOAA dismissed stale reviews from JohnSteffen-NOAA and apchoiCMD via 1cf85fb February 26, 2025 20:51

AndrewEichmann-NOAA requested a review from guillaumevernieres February 26, 2025 20:51

guillaumevernieres approved these changes Feb 27, 2025

View reviewed changes

guillaumevernieres merged commit 0a1a96c into develop Feb 27, 2025

guillaumevernieres deleted the feature/insitu-concat branch February 27, 2025 15:53

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Incorporate multi-cycle concatination and separation of variables in post-BUFR observations#1504

Incorporate multi-cycle concatination and separation of variables in post-BUFR observations#1504
guillaumevernieres merged 16 commits into
developfrom
feature/insitu-concat

AndrewEichmann-NOAA commented Feb 20, 2025 •

edited

Loading

Uh oh!

guillaumevernieres left a comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emcbot commented Feb 21, 2025

Uh oh!

AndrewEichmann-NOAA commented Feb 24, 2025

Uh oh!

emcbot commented Feb 24, 2025

Uh oh!

apchoiCMD left a comment

Uh oh!

Uh oh!

AndrewEichmann-NOAA commented Feb 26, 2025

Uh oh!

apchoiCMD commented Feb 26, 2025

Uh oh!

guillaumevernieres left a comment

Uh oh!

Uh oh!

guillaumevernieres left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

Conversation

AndrewEichmann-NOAA commented Feb 20, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Companion PRs

Issues

Automated CI tests to run in Global Workflow

Uh oh!

guillaumevernieres left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

emcbot commented Feb 21, 2025

Uh oh!

AndrewEichmann-NOAA commented Feb 24, 2025

Uh oh!

emcbot commented Feb 24, 2025

Uh oh!

apchoiCMD left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

AndrewEichmann-NOAA commented Feb 26, 2025

Uh oh!

apchoiCMD commented Feb 26, 2025

Uh oh!

guillaumevernieres left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

guillaumevernieres left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

AndrewEichmann-NOAA commented Feb 20, 2025 •

edited

Loading