Skip to content

IODA Converter with In-Situ Observations Concatenation and ObsError Inflation#1472

Merged
guillaumevernieres merged 15 commits into
developfrom
feature/redate-insitu
Feb 13, 2025
Merged

IODA Converter with In-Situ Observations Concatenation and ObsError Inflation#1472
guillaumevernieres merged 15 commits into
developfrom
feature/redate-insitu

Conversation

@apchoiCMD

@apchoiCMD apchoiCMD commented Jan 29, 2025

Copy link
Copy Markdown
Collaborator

Description

This PR shows that an IODA Converter with Marine In-Situ Observations Concatenation and ObsError Inflation Capabilities from the table that shows various types of in-situ obs ("x" means that the variable is not provided)
Type Variables Argo Bathy Glider Tesac Tropical xbtctd drifter trkob
MetaData dateTime - - - - - - - -
- depth - - - - - - $\color{red}{\textsf{x}}$ $\color{red}{\textsf{x}}$
- latitude - - - - - - - -
- longitude - - - - - - - -
- oceanbasin - - - - - - - -
ObsError waterTemperature
salinity
-
-
-
$\color{red}{\textsf{x}}$
- - - - -
$\color{red}{\textsf{x}}$
-
ObsValue waterTemperature
salinity
-
-
-
$\color{red}{\textsf{x}}$
- - - - -
$\color{red}{\textsf{x}}$
-
PreQC waterTemperature
salinity
-
-
-
$\color{red}{\textsf{x}}$
- - - - -
$\color{red}{\textsf{x}}$
-
  • Double check on sail drone and axbt in-situ obs soon
  • Input yaml will be updated once test inputs are ready

Companion PRs

Nope

Issues

Resolves #1465

Automated CI tests to run in Global Workflow

  • atm_jjob
  • C96C48_ufs_hybatmDA
  • C96C48_hybatmaerosnowDA
  • C48mx500_3DVarAOWCDA
  • C48mx500_hybAOWCDA
  • C96C48_hybatmDA

Comment thread utils/obsproc/InsituAll2ioda.h Outdated
@apchoiCMD

Copy link
Copy Markdown
Collaborator Author

Result is here by using ARGO @guillaumevernieres @AndrewEichmann-NOAA @JohnSteffen-NOAA

- gdas.t00z.insitu_profile_argo.2021032400.nc4
- gdas.t06z.insitu_profile_argo.2021032406.nc4
- gdas.t12z.insitu_profile_argo.2021032412.nc4
- gdas.t18z.insitu_profile_argo.2021032418.nc4
- gdas.t00z.insitu_profile_argo.2021032500.nc4
- gdas.t06z.insitu_profile_argo.2021032506.nc4
- gdas.t12z.insitu_profile_argo.2021032512.nc4
- gdas.t18z.insitu_profile_argo.2021032518.nc4
- gdas.t00z.insitu_profile_argo.2021032600.nc4

insitu_argo_redate_cycle

@apchoiCMD apchoiCMD marked this pull request as ready for review January 30, 2025 21:54
Comment thread utils/obsproc/Rads2Ioda.h
Comment thread utils/obsproc/InsituAll2ioda.h
Comment thread utils/test/testinput/gdas_insituall2ioda.yaml

@guillaumevernieres guillaumevernieres left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see comments

Comment thread utils/obsproc/InsituAll2ioda.h
Comment thread utils/obsproc/InsituAll2ioda.h Outdated
Comment thread utils/obsproc/InsituAll2ioda.h Outdated
Comment thread utils/obsproc/InsituAll2ioda.h Outdated
@guillaumevernieres guillaumevernieres added the hercules-GW-RT Queue for automated testing with global-workflow on Hercules label Feb 11, 2025
@emcbot emcbot added hercules-GW-RT-Running Automated testing with global-workflow running on Hercules and removed hercules-GW-RT Queue for automated testing with global-workflow on Hercules labels Feb 11, 2025
@emcbot

emcbot commented Feb 11, 2025

Copy link
Copy Markdown

Automated GW-GDASApp Testing Results:
Machine: hercules

Start: Tue Feb 11 16:03:27 CST 2025 on hercules-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Tue Feb 11 16:42:19 CST 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp -E atm_jjob|C96C48_ufs_hybatmDA|C96C48_hybatmaerosnowDA|C48mx500_3DVarAOWCDA|C48mx500_hybAOWCDA|C96C48_hybatmDA
Tests:                                  *Failed*
Tests: Failed at Tue Feb 11 16:50:42 CST 2025
Tests: 97% tests passed, 1 tests failed out of 33
Tests: see output at /work2/noaa/da/role-da/CI/hercules/GDASApp/workflow/PR/1472/global-workflow/sorc/gdas.cd/build/log.ctest

@emcbot emcbot added hercules-GW-RT-Failed Automated testing with global-workflow failed on Hera and removed hercules-GW-RT-Running Automated testing with global-workflow running on Hercules labels Feb 11, 2025
@RussTreadon-NOAA

Copy link
Copy Markdown
Contributor

Hercules CI failure

Check log.ctest. test_gdasapp_util_insituall2ioda failed. Manually rerun this test with verbose output. Failure due to missing input yaml.

    Start 1598: test_gdasapp_util_insituall2ioda

1598: Test command: /work2/noaa/da/role-da/CI/hercules/GDASApp/workflow/PR/1472/global-workflow/sorc/gdas.cd/build/bin/gdas_obsprovider2ioda.x "../testinput/gdas_insituall2ioda.yaml"
1598: Environment variables:
1598:  OMP_NUM_THREADS=1
1598: Test timeout computed to be: 1500
1598: Exception: Cannot open ../testinput/gdas_insituall2ioda.yaml  (No such file or directory)
1598: terminate called after throwing an instance of 'eckit::CantOpenFile'
1598:   what():  Cannot open ../testinput/gdas_insituall2ioda.yaml  (No such file or directory)
1/1 Test #1598: test_gdasapp_util_insituall2ioda ...Subprocess aborted***Exception:  12.96 sec

@guillaumevernieres

Copy link
Copy Markdown
Contributor

Thanks @RussTreadon-NOAA .
@apchoiCMD , you probably forgot to create the link to the test yaml in the cmakelists.txt?

@apchoiCMD

Copy link
Copy Markdown
Collaborator Author

@RussTreadon-NOAA @guillaumevernieres Thanks for heads up and I will look into it as soon as possible.

@apchoiCMD apchoiCMD added the hercules-GW-RT Queue for automated testing with global-workflow on Hercules label Feb 12, 2025
@emcbot emcbot added hercules-GW-RT-Running Automated testing with global-workflow running on Hercules and removed hercules-GW-RT Queue for automated testing with global-workflow on Hercules labels Feb 12, 2025
@emcbot

emcbot commented Feb 12, 2025

Copy link
Copy Markdown

Automated GW-GDASApp Testing Results:
Machine: hercules

Start: Wed Feb 12 09:20:13 CST 2025 on hercules-login-1.hpc.msstate.edu
---------------------------------------------------
Build:                                 *SUCCESS*
Build: Completed at Wed Feb 12 09:57:04 CST 2025
---------------------------------------------------
Tests: ctest -j12 -R gdasapp -E atm_jjob|C96C48_ufs_hybatmDA|C96C48_hybatmaerosnowDA|C48mx500_3DVarAOWCDA|C48mx500_hybAOWCDA|C96C48_hybatmDA
Tests:                                 *SUCCESS*
Tests: Completed at Wed Feb 12 10:02:08 CST 2025
Tests: 100% tests passed, 0 tests failed out of 33

@emcbot emcbot added hercules-GW-RT-Passed Automated testing with global-workflow successful on Hercules and removed hercules-GW-RT-Running Automated testing with global-workflow running on Hercules labels Feb 12, 2025

@guillaumevernieres guillaumevernieres left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@guillaumevernieres guillaumevernieres removed the hercules-GW-RT-Failed Automated testing with global-workflow failed on Hera label Feb 13, 2025
@guillaumevernieres guillaumevernieres merged commit d5083d5 into develop Feb 13, 2025
@guillaumevernieres guillaumevernieres deleted the feature/redate-insitu branch February 13, 2025 14:13
danholdaway added a commit that referenced this pull request Mar 26, 2025
* develop:
  added original date (#1570)
  Update OCEAN_BASIN_FILE_ORION path for marine ctest (#1569)
  Fixing window_begin and end in yaml for redating ADT (#1565)
  Save LETKF mean analysis and mean increment for verification (#1566)
  Update JEDI hashes (20250324) (#1567)
  Removes obsolete json files from marine bufr2ioda (#1564)
  Add hybrid 4denvar as a phase3 validation configuration (#1552)
  Bugfix/ocn ecen (#1550)
  Bugfix SSS obs for LETKF (#1560)
  Bugfix ocean ensemble recentering (#1558)
  apply ci path patch only when running g-w ctests (#1554)
  Update JEDI hashes (20250317)  (#1531)
  add option to create_experiment.py in create_exp.sh  (#1546)
  Update jcb hashes and support initial IODA stats capability (#1547)
  Fix GW-CI CTest dependencies (#1534)
  Converts prepoceanobs to use JCB-style obs list and config files (#1527)
  JEDI-based ensemble recentering and analysis calculation (#1488)
  tools for phase3 validation (#1526)
  Update JEDI hashes (20250306) (#1520)
  New jcb-gdas # and insitu obs prep bug fix  (#1523)
  Marine in situ obs update (SSS, moorings, drifters) (#1511)
  Save marine background and analysis spread (#1517)
  Minor build and install updates and ctest cleanup (#1512)
  correct logic for upper loop limit in chem_diagb (#1516)
  Incorporate multi-cycle concatination and separation of variables in post-BUFR observations (#1504)
  Update JEDI hashes (20250225) (#1475)
  Add a depth dependent e-fold scale (#1494)
  Update jcb and da-utils hashes to support land DA. (#1509)
  Add options to inflate the ensemble while recentering (#1501)
  Add components to use jcb yamls for marine letkf job (#1486)
  Add GNSSRO bufr2ioda and json files (#1448)
  Removal of the marine verification tools (#1499)
  update logic to identify MSU machine (#1498)
  IODA Converter with In-Situ Observations Concatenation and ObsError Inflation (#1472)
  Add sea ice recentering (#1478)
  marine vrfy bugfix (#1483)
  add LD_LIBRARY_PATH patch for wcoss2 build (#1484)
  Bugfix for ensemble recenter + add a test (#1477)
  Add code changes for the implementation of the  low-resolution B (#1441)
  update spack-stack path in Hera modulefiles (#1471)
  fixed glider output filename (#1469)
  add gdas_waveinit to C48mx500_3DVarAOWCDA ctest suite (#1470)
  update jcb-gdas hash 20250128 (#1466)
  Time series of omb stats and html doc (#1463)
  Update marine vrfy HTML codes (#1464)
  Automate the verify tools to submit from start pdy to end pdy (#1458)
  Update an offline marine vrfy tool (#1447)
  Update JEDI hashes (20250127) (#1460)
DavidNew-NOAA pushed a commit that referenced this pull request Jan 16, 2026
…nflation (#1472)

Concatenation and re-dating of ioda files.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

hercules-GW-RT-Passed Automated testing with global-workflow successful on Hercules

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Redating and Error amplification for nc4 In-Situ obs

6 participants