Skip to content

Add BUFR2IODA python API converter to prepoceanobs task#914

Merged
guillaumevernieres merged 46 commits into
developfrom
feature/marine_addBUFR2IODAconverter
Mar 28, 2024
Merged

Add BUFR2IODA python API converter to prepoceanobs task#914
guillaumevernieres merged 46 commits into
developfrom
feature/marine_addBUFR2IODAconverter

Conversation

@ShastriPaturi
Copy link
Copy Markdown
Collaborator

@ShastriPaturi ShastriPaturi commented Feb 8, 2024

This PR adds BUFR2IODA python API converter to prepoceanobs task and does the following:

  1. Subsampled monthly bathy, tesac and trkob in situ BUFR files are tested.
  2. The subsampled files have been renamed (from .201804.dcom_subsampled) as .tz..tm00.bufr_d
  3. Runs for one cycle: 2018041512.

closes #892

Overlapping windows application will be added in a subsequent PR

AndrewEichmann-NOAA
AndrewEichmann-NOAA reviewed Feb 8, 2024
View reviewed changes
Comment thread scripts/exglobal_prep_ocean_obs.py Outdated
Comment thread parm/soca/obsprep/obsprep_config.yaml Outdated
Comment thread scripts/exglobal_prep_ocean_obs.py Outdated
@ShastriPaturi ShastriPaturi mentioned this pull request Feb 15, 2024
Closed
AndrewEichmann-NOAA
AndrewEichmann-NOAA reviewed Feb 20, 2024
View reviewed changes
Copy link
Copy Markdown
Collaborator

@AndrewEichmann-NOAA AndrewEichmann-NOAA left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do the additions to ush/ioda - I guess the bufr2ioda converters - already exist in a different repo? I wonder if we can import them from there

Comment thread scripts/exglobal_prep_ocean_obs.py Outdated
Comment thread scripts/exglobal_prep_ocean_obs.py
@emcbot emcbot added the orion-GW-RT-Failed Automated testing with global-workflow failed on Orion label Mar 26, 2024
@AndrewEichmann-NOAA
Copy link
Copy Markdown
Collaborator

AndrewEichmann-NOAA commented Mar 27, 2024

I started the CI forgetting about the outstanding g-w build issue on Rocky 8 - gdasapp at least builds:

build_gfs_utils.sh completed successfully!
build_ufs_utils.sh failed with status 2!
build_ww3prepost.sh failed with status 2!
build_gsi_monitor.sh completed successfully!
build_ufs.sh failed with status 2!
build_gsi_utils.sh completed successfully!
build_upp.sh failed with status 2!
build_gdas.sh completed successfully!
BUILD ERROR: One or more components failed to build
  Check the associated build log(s) for details.

@AndrewEichmann-NOAA
Copy link
Copy Markdown
Collaborator

AndrewEichmann-NOAA commented Mar 27, 2024 •
edited
Loading

@guillaumevernieres @CoryMartin-NOAA @ShastriPaturi The branch as is builds and completes all the global-workflow ctests (besides vrfy) on Hera while delivering populated in-situ obs files to the ROTDIR. I've changed the ctests so that this can run without the associated PR to global-workflow.

@guillaumevernieres guillaumevernieres self-requested a review March 28, 2024 13:16
@guillaumevernieres guillaumevernieres added hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion and removed hera-GW-RT-Failed Automated testing with global-workflow failed on Hera orion-GW-RT-Failed Automated testing with global-workflow failed on Orion labels Mar 28, 2024
@emcbot emcbot added hera-GW-RT-Running Automated testing with global-workflow running on Hera orion-GW-RT-Running Automated testing with global-workflow running on Orion and removed hera-GW-RT Queue for automated testing with global-workflow on Hera orion-GW-RT Queue for automated testing with global-workflow on Orion labels Mar 28, 2024
@emcbot
Copy link
Copy Markdown

emcbot commented Mar 28, 2024

Automated Global-Workflow GDASApp Testing Results:
Machine: hera

Start: Thu Mar 28 14:00:35 UTC 2024 on hfe12
---------------------------------------------------
Build:                                  *FAILED*
Build: Failed at Thu Mar 28 14:51:22 UTC 2024
Build: see output at /scratch1/NCEPDEV/da/Cory.R.Martin/CI/GDASApp/workflow/PR/914/global-workflow/sorc/log.build

@emcbot emcbot added hera-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hera-GW-RT-Running Automated testing with global-workflow running on Hera labels Mar 28, 2024
@RussTreadon-NOAA
Copy link
Copy Markdown
Contributor

RussTreadon-NOAA commented Mar 28, 2024

Hera GW-RT cloned g-w develop at f0b912be. This is an older hash. We should clone g-w develop at d6be3b5. Unfortunately even with this change GW-RT will still fail during the build step. upp.fd still does not build on Hera Rocky-8.

Here's the build log from g-w develop at d6be3b5

Running "module reset". Resetting modules to system default. The following $MODULEPATH directories have been removed: None
Building gsi_enkf, ufs, gfs_utils, gdas, ww3prepost, ufs_utils, gsi_utils, gsi_monitor, upp
Starting build_gsi_enkf.sh
Starting build_ufs.sh
Starting build_gfs_utils.sh
Starting build_gdas.sh
Starting build_ww3prepost.sh
Starting build_ufs_utils.sh
Starting build_gsi_utils.sh
Starting build_gsi_monitor.sh
Starting build_upp.sh
build_gsi_enkf.sh completed successfully!
build_gfs_utils.sh completed successfully!
build_ufs_utils.sh completed successfully!
build_gsi_utils.sh completed successfully!
build_gsi_monitor.sh completed successfully!
build_ww3prepost.sh completed successfully!
build_upp.sh failed with status 2!
build_ufs.sh completed successfully!
build_gdas.sh completed successfully!
BUILD ERROR: One or more components failed to build
  Check the associated build log(s) for details.

@emcbot
Copy link
Copy Markdown

emcbot commented Mar 28, 2024

Automated Global-Workflow GDASApp Testing Results:
Machine: orion

Start: Thu Mar 28 09:01:07 CDT 2024 on Orion-login-1.HPC.MsState.Edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Thu Mar 28 10:07:59 CDT 2024
---------------------------------------------------
Tests:                                 *SUCCESS*
Tests: Completed at Thu Mar 28 10:38:24 CDT 2024
Tests: 100% tests passed, 0 tests failed out of 53

@emcbot emcbot added orion-GW-RT-Passed Automated testing with global-workflow successful on Orion and removed orion-GW-RT-Running Automated testing with global-workflow running on Orion labels Mar 28, 2024
guillaumevernieres
guillaumevernieres approved these changes Mar 28, 2024
View reviewed changes
Copy link
Copy Markdown
Contributor

@guillaumevernieres guillaumevernieres left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@guillaumevernieres guillaumevernieres merged commit 33b4cb9 into develop Mar 28, 2024
danholdaway added a commit that referenced this pull request Apr 8, 2024
@danholdaway
Merge remote-tracking branch 'origin/develop' into feature/use_jcb_atm
b256b0f 
* origin/develop:
  Use <filesystem> on a non c++17 supported machine (WCOSS ACORN) (#1026)
  Change generate_com to declare_from_tmpl (#1025)
  Commenting out more of the marine bufr 2 ioda stuff (#1018)
  make driver consistent with workflow driver (#1016)
  Update hashes now that GSI-B is working for EnVar (#1015)
  Add GitHub CLI to path for CI (#1014)
  Use _anl rather than _ges dimensions for increments in FV3 increment converter YAML (#1013)
  Fix inconsistent VIIRS preprocessing test (#1012)
  remove gdas_ prefix from executable filename in test_gdasapp_fv3jedi_fv3inc (#1010)
  Bugfix on Broken GHRSST Ioda Converter (#1004)
  Moved the marine converters to a "safe" place (#1007)
  restore ATM local ensemble ctest functionality (#1003)
  Add BUFR2IODA python API converter to prepoceanobs task (#914)
  Remove sst's from obs proc (#1001)
  JEDI increment write to cubed sphere history (#983)
  [End- to End Test code sprint] Add SEVIRI METEOSAT-8 and METEOSAT-11 to end-to-end testing (#766)
@guillaumevernieres guillaumevernieres deleted the feature/marine_addBUFR2IODAconverter branch April 16, 2024 19:15
DavidNew-NOAA pushed a commit that referenced this pull request Jan 16, 2026
@ShastriPaturi @guillaumevernieres
@AndrewEichmann-NOAA @CoryMartin-NOAA
Add BUFR2IODA python API converter to prepoceanobs task (#914)
dfc7273 
This PR adds BUFR2IODA python API converter to prepoceanobs task and
does the following:
1. Subsampled monthly bathy, tesac and trkob in situ BUFR files are
tested.
2. The subsampled files have been renamed (from
<obtype>.201804.dcom_subsampled) as <cycle>.t<hh>z.<obtype>.tm00.bufr_d
3. Runs for one cycle: 2018041512.

closes #892 

Overlapping windows application will be added in a subsequent PR

---------

Co-authored-by: Guillaume Vernieres <guillaume.vernieres@noaa.gov>
Co-authored-by: AndrewEichmann-NOAA <andrew.eichmann@noaa.gov>
Co-authored-by: AndrewEichmann-NOAA <58948505+AndrewEichmann-NOAA@users.noreply.github.com>
Co-authored-by: Cory Martin <cory.r.martin@noaa.gov>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@guillaumevernieres guillaumevernieres guillaumevernieres approved these changes

@apchoiCMD apchoiCMD Awaiting requested review from apchoiCMD

@AndrewEichmann-NOAA AndrewEichmann-NOAA Awaiting requested review from AndrewEichmann-NOAA

@CoryMartin-NOAA CoryMartin-NOAA Awaiting requested review from CoryMartin-NOAA

Assignees

No one assigned

Labels

hera-GW-RT-Failed Automated testing with global-workflow failed on Hera orion-GW-RT-Passed Automated testing with global-workflow successful on Orion

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Make BUFRI2IODA conversion part of ocean prep obs task

6 participants

@ShastriPaturi @emcbot @AndrewEichmann-NOAA @RussTreadon-NOAA @CoryMartin-NOAA @guillaumevernieres