SPRINT: IODA-converters staging branch by rhoneyager · Pull Request #817 · JCSDA-internal/ioda-converters

rhoneyager · 2022-03-07T20:16:17Z

Description

This is the staging branch for the IODA-converters sprint of March 2022. This PR holds IODA-converters changes for individual converters.

When you create a PR in Task 1 of the sprint, you should make a PR targeting this branch of the code.

For Task 2, your changes can target the develop branch directly.

Errata

In this branch, the IODA validator is set to enforcing mode. This will help us determine which converters are still failing in CI.

Note: due to the CI system logic, your individual PRs will still be run in permissive mode. The CI tests will all pass. Only the feature/sprint-ioda-converters branch will log validation test failures.

rhoneyager · 2022-03-08T17:02:11Z

CI/buildspec_clang.yml


     # ioda
-      - ./clone.sh $GIT_USER $GIT_PASS jcsda-internal/ioda $CODEBUILD_GIT_BRANCH ioda /jcsda/ioda-bundle develop
+      - ./clone.sh $GIT_USER $GIT_PASS jcsda-internal/ioda $CODEBUILD_GIT_BRANCH ioda /jcsda/ioda-bundle feature/sprint-ioda-converters


Note to self: before merging to develop, the buildspec_* files should be reverted.

…ng conventions (#838) * clean up code a lot while also moving to JEDI naming conventions * updated output test-reference file * resolve various small coding_norms * tiny change to global attr * eliminate datetimeReference global attr * updated test-ref file to remove datetimeReference global attr * add comment to preQc value of 2 plus bring back datetimeReference attribute * take away global attribute of Location * Updated testoutput file for Greg Co-authored-by: gthompsnJCSDA <gthompsn@ucar.edu> Co-authored-by: Ryan Honeyager <honeyage@ucar.edu>

* Print existing datetime var * Convert ISO string to dt object * Remove extra boolean * Make error handling work Co-authored-by: Ryan Honeyager <honeyage@ucar.edu>

* fix a host of IODA conention issues to bring in compliance with v2 * revert last commit by using original file again * initial try to bring converter up to IODA v2 standards * put time into datetimeRange attribute for now * removed unused large ATMS file * add the new ATMS output with the latest IODA conventions in place * update the satelliteIdentifier to not have units * fix whitespace coding norm error * change dateTime to seconds since 1970 epoch and nlocs dimension to int32 * add the new output file for the ctest * add sensor WMO ID as a string and use group names inherited from IODA * update the testoutput file for atms decode * move the satellite platform to a global attribute * reverse the conditional to make it less confusing fix coding norm issue * fix a host of IODA conention issues to bring in compliance with v2 * revert last commit by using original file again * again try to add the correct output file * add nested list for channel dimension * add final changes to align with current features in the IODA interface * change use of datetimeRange to datetimeReference * while we are at it add the datetimeRange from the data itself * put and error exit if call tries to merge satellites into single IODA file * Updated test reference with proper UTC dateTime values. * Updated test reference for atms test. Co-authored-by: Ryan Honeyager <honeyage@ucar.edu> Co-authored-by: Stephen Herbener <stephen.herbener@gmail.com>

…ta Conventions (#856) * allow units attribute to a variable to be optional * allow units attribute to a variable to be optional * skip units if set to none or empty quotes * updates to aircar, airep, amdar to match naming conventions * abbeviate degree for windDirection * fix up windDirection and couple other items * many yaml files changed for JEDI naming conventions * adjust many files to align with JEDI naming conventions * make all new test ref files for JEDI naming conventions during sprint Co-authored-by: gthompsnJCSDA <gthompsn@ucar.edu> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com>

…o follow JEDI Data Conventions (#868) * allow units attribute to a variable to be optional * allow units attribute to a variable to be optional * skip units if set to none or empty quotes * updates to aircar, airep, amdar to match naming conventions * abbeviate degree for windDirection * fix up windDirection and couple other items * many yaml files changed for JEDI naming conventions * adjust many files to align with JEDI naming conventions * make all new test ref files for JEDI naming conventions during sprint * changed to align with JEDI Data Conventions * serious changes to adhere to JEDI Data Conventions * update testref files of a bunch of bufr2ioda.x output files for JEDI Data Conventions * fix a few files to match testref Co-authored-by: gthompsnJCSDA <gthompsn@ucar.edu> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: Ryan Honeyager <honeyage@ucar.edu>

#859) * Modify gsidiag converter to follow JEDI data convention and update the test reference. * Code clean up * Remove debugging line * Fix platform and sensor type from string to integer and update test output * Remove writing the attribute for dateTime and update test reference output * Perform unit change for frequency and wavenumber to meet JEDI data convention and update output test reference * Fix unit for one derived variable and update output test reference * Remove debugging and commented outlines Co-authored-by: Ryan Honeyager <honeyage@ucar.edu>

* Modified cryosat_ice2ioda.py and emc_ice2ioda.py to comply with code converntion and updated testoutput files. * rerun build, make and ctest, updated testoutput/cryosat2_L2.nc and emc_ice_ioda2.nc Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: Ryan Honeyager <honeyage@ucar.edu>

…w) to follow the JEDI Data Conventions (#863) * change AMDAR converter to align with JEDI data conventions * change Buoy converter to align with JEDI data conventions * change Ship converter to align with JEDI data conventions * change SYNOP converter to align with JEDI data conventions * eliminate units when they are supposed to be missing Co-authored-by: gthompsnJCSDA <gthompsn@ucar.edu> Co-authored-by: Ryan Honeyager <honeyage@ucar.edu>

…erter code spring (#854) * Modified ghcn_snod2ioda.py and updated testoutput * Updated smos_ssm2ioda.py and output results * Update src/land/smap_ssm2ioda.py and testoutput * update src/land/imsfv3_scf2ioda.py and testoutput * update src/land/owp_snow_obs.py and testoutput result * Update smap9km_ssm2ioda.py and corresponding result * Fix satellite and sensor ID issue and update the results * Address the issues araised by Cory Martin except for dateTime * Fixed two wrong spelling in smap_ssm2ioda.py * Update resulst for INIT64 for Location * update copyright time period * Modified str datetime into dataTime and the results. * Removed one comment line * modified filling vale for smap9km_ssm2ioda.py * Fixed datetime with dateTime and updated the result * Changed station_id to stationIdentification and update the result Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com>

* updates so far * argoClim2ioda.py updates * update testoutput because of dateTime int changes * update to testoutput Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: Ryan Honeyager <honeyage@ucar.edu>

…ters to follow JEDI data convention (#858) * Marine smos_sss and smap_sss converters update * Marine/smos_sss2ioda.py and Marine/smap_sss2ioda.py * remove commented out lines from smos_sss and smap_sss converters * smos, smap, and hgodas converters update * testoutput/ nc file updates * Fixed double-wording in smap and ssmos converters * Restart CI Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: Ryan Honeyager <honeyage@ucar.edu>

* Changes regarding iodaconv * Update chem/modis_aod for JEDI IODA convention * Resolove PreQC unit issues * significant changes, cleaning up, etc. * Update testoutput/modis_aod.nc after Greg's change * Update modis aod test reference files Co-authored-by: Greg Thompson <63315545+gthompsnJCSDA@users.noreply.github.com> Co-authored-by: gthompsnJCSDA <gthompsn@ucar.edu> Co-authored-by: Ryan Honeyager <honeyage@ucar.edu>

* Removed two land converters with grib input files * Removed two land conveters from the test/CMakeLists.txt * Removed the input and output files for two land converters

…n with JEDI Data Conventions (#874) * allow units attribute to a variable to be optional * allow units attribute to a variable to be optional * skip units if set to none or empty quotes * updates to aircar, airep, amdar to match naming conventions * abbeviate degree for windDirection * fix up windDirection and couple other items * many yaml files changed for JEDI naming conventions * adjust many files to align with JEDI naming conventions * make all new test ref files for JEDI naming conventions during sprint * start of changes for matching JEDI Data Conventions * able to combine v2 into prior MHS file * able to merge v2 yaml into HRS earlier file when updating to JEDI Data Conventions * update testref files compliant with JEDI Data Conventions * new addition of AVHRR testinput/output files and consistent with JEDI Data Conventions * new addition of AVHRR testinput/output files and consistent with JEDI Data Conventions * add ctest for AVHRR * new addition of AIRS testinput/output yaml and ctest consistent with JEDI Data Conventions * new addition of AMSUA testinput/output yaml and ctest consistent with JEDI Data Conventions * new addition of ATMS testinput/output yaml and ctest consistent with JEDI Data Conventions * new addition of CRIS testinput/output yaml and ctest consistent with JEDI Data Conventions * new addition of SEVIRI (CSR) testinput/output yaml and ctest consistent with JEDI Data Conventions * new addition of SSMIS testinput/output yaml and ctest consistent with JEDI Data Conventions * new addition of IASI testinput/output yaml and ctest consistent with JEDI Data Conventions * remove the v2 reference of filename because we will be using JEDI Data Conventions eventually * eliminate channelBlock, scaleFactor, and start/end channel info for now * fix up the test ref files * major reduction in file sizes * fix test filenames and such * add in a missing test ref file Co-authored-by: gthompsnJCSDA <gthompsn@ucar.edu> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: Ryan Honeyager <honeyage@ucar.edu>

* allow units attribute to a variable to be optional * allow units attribute to a variable to be optional * skip units if set to none or empty quotes * updates to aircar, airep, amdar to match naming conventions * abbeviate degree for windDirection * fix up windDirection and couple other items * many yaml files changed for JEDI naming conventions * adjust many files to align with JEDI naming conventions * make all new test ref files for JEDI naming conventions during sprint * start of changes for matching JEDI Data Conventions * able to combine v2 into prior MHS file * able to merge v2 yaml into HRS earlier file when updating to JEDI Data Conventions * update testref files compliant with JEDI Data Conventions * new addition of AVHRR testinput/output files and consistent with JEDI Data Conventions * new addition of AVHRR testinput/output files and consistent with JEDI Data Conventions * add ctest for AVHRR * new addition of AIRS testinput/output yaml and ctest consistent with JEDI Data Conventions * new addition of AMSUA testinput/output yaml and ctest consistent with JEDI Data Conventions * new addition of ATMS testinput/output yaml and ctest consistent with JEDI Data Conventions * new addition of CRIS testinput/output yaml and ctest consistent with JEDI Data Conventions * new addition of SEVIRI (CSR) testinput/output yaml and ctest consistent with JEDI Data Conventions * new addition of SSMIS testinput/output yaml and ctest consistent with JEDI Data Conventions * new addition of IASI testinput/output yaml and ctest consistent with JEDI Data Conventions * remove the v2 reference of filename because we will be using JEDI Data Conventions eventually * eliminate channelBlock, scaleFactor, and start/end channel info for now * fix up the test ref files * major reduction in file sizes * fix test filenames and such * add in a missing test ref file * revert back to develop version of MHS file * fix the test ref files for splitting, filtering, and filter_split * update another test ref file for bufr_mhs2ioda * ensuring updated test ref file * fixing input and output files (again) Co-authored-by: gthompsnJCSDA <gthompsn@ucar.edu> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: Ryan Honeyager <honeyage@ucar.edu>

srherbener · 2022-05-13T17:14:38Z

@YoulongXia-NOAA, updating the test reference got the smap9k test to pass, but now the validator fails on the smap9k reference file:

Reading YAML from /jcsda/ioda-bundle/ioda/share/ioda/yaml/validation/ObsSpace.yaml
Processing data file: /jcsda/ioda-bundle/iodaconv/test/testoutput/smap9km_ssm.nc
 Verifying that all required groups exist
 Verifying group MetaData
 Verifying group ObsValue
 Verifying group ObsError
 Verifying group PreQC
 Verifying group /
 Verifying dimension names
 Verifying variable information
  Variable MetaData/dateTime
  Variable MetaData/latitude
  Variable MetaData/longitude
  Variable MetaData/vegetationOpacity
   Warning: Attribute 'units' is present but is not listed as a required or optional attribute in the spec.
   ERROR: Variable 'MetaData/vegetationOpacity' needs units, but the 'units' attribute does not exist in the YAML.
  Variable MetaData/earthSurfaceType
  Variable MetaData/easeRowIndex
   Warning: Variable 'MetaData/easeRowIndex' is not listed in the YAML conventions file.
  Variable MetaData/easeColumnIndex
   Warning: Variable 'MetaData/easeColumnIndex' is not listed in the YAML conventions file.
  Variable ObsValue/soilMoistureVolumetric
  Variable ObsError/soilMoistureVolumetric
  Variable PreQC/soilMoistureVolumetric
Final results:
  # errors:      1
  # warnings:    3

CTEST_FULL_OUTPUT

Can you help get this fixed? Thanks!

YoulongXia-NOAA · 2022-05-13T17:46:09Z

vegetationOpacity

@srherbener, I checked both ObsSpace.yaml and convention table, vegetationOpacity has no units. you need to remove
self.varAttrs[('vegetationOpacity', 'MetaData')]['units'] = 'unitless' from smap9km_ssm2ioda.py. I forgot remove it.

For MetaData/easeRowIndex and MetaData/easeColumnIndex, I need to add them to ObsSpace.yaml . I will work them soon.

YoulongXia-NOAA · 2022-05-13T17:49:13Z

@srherbener, by the way, if I update ObsSpace.yaml , do I need to use feature/sprint-ioda-converters or develop as the reference branch? After I get your response, I will work it soon.

srherbener · 2022-05-13T20:34:12Z

vegetationOpacity

@srherbener, I checked both ObsSpace.yaml and convention table, vegetationOpacity has no units. you need to remove self.varAttrs[('vegetationOpacity', 'MetaData')]['units'] = 'unitless' from smap9km_ssm2ioda.py. I forgot remove it.

For MetaData/easeRowIndex and MetaData/easeColumnIndex, I need to add them to ObsSpace.yaml . I will work them soon.

@YoulongXia-NOAA I have pushed to github a new feature brach, feature/sprint-smap9k-cleanup, with the fix for the vegetationOpacity variable (ie, I removed the units attribute from this variable). I've updated the test reference file accordingly and we are now down to the two warnings from the validator (which are the two new variables, easeRowIndex and easeColumnIndex).

The new feature/sprint-smap9k-cleanup branch is based on the feature/sprint-ioda-converters branch. Please make your additions to ObsSpace.yaml on the new feature/sprint-smap9k-cleanup branch and then submit a PR to the feature/sprint-ioda-converters branch. Thanks!

YoulongXia-NOAA · 2022-05-13T20:38:39Z

@srherbener, okay, got it. Than you.

…the conventions specification. (#946)

srherbener · 2022-05-27T14:37:00Z

@FanHan-NOAA I think that the conflict needs to be resolved by making the same changes in the test reference file test/testoutput/sfc_tv_obs_2018041500.nc4 that were done in #953. Thanks!

* Add files via upload * Update file ran via feature/sprint-ioda-converters * Delete sfc_tv_obs_2018041500.nc4 * Add files via upload * Updated test reference for surface virtual temperature test for the gsi ncdiag converter Co-authored-by: Stephen Herbener <stephen.herbener@gmail.com>

* WIP: getting the converter library updated with latest ioda python API fixes. * WIP: Fixed cases where direct assinment of int64 types to dateTime were crashing when setting the fill value. * Updated test reference files for those tests that failed due to only the dateTime fill value not matching.

…le to decipher them (#961) * nearly working updated code to handle subsets of sondes in single BUFR msg * more debugging * continue debugging * effort continues * very very close to working; still getting an abort with no year/month/day problem * a few more bug fixes after testing on more real-time data * change some debug printing * light changes for coding_norms * make comments about data bounds per review request * attempting to resolve dumb merge conflict * BUFR lib name fixes * temporary delete of test/testoutput/satwinds_ssec * tiny changes per PR requests * update reference file * coding norms fix Co-authored-by: gthompsnJCSDA <gthompsn@ucar.edu> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: Ryan Honeyager <honeyage@ucar.edu>

…ing up fill values instead of the develop version of that code.

…ible for November code sprint (#1119) * Add ability to capture useful radiosonde QC info (#1015) * add in important QC info for sondes * newly updated testoutput file after changes * need to abort out of BUFR messages that claim to contain compressed data Co-authored-by: gthompsnJCSDA <gthompsn@ucar.edu> * Bugfix/omi ompsnm converter get files correctly at beginning of window (#1017) * fix to grab files at start of the window. * fix missing space. Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> * Add a new group in the IODA format for L2 products with averaging kernel and slant columns (#1018) * firt commit, add smoothing group * group smoothing in lib * correct changes * restore * add smoothing for mopitt * test data * change group name * update test data Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> * Feature/modis aod from h4 (#942) * change modis aod converter to take antive hdf4 files as input this requires installing the pyhdf python package * update test ref:wq * fix coding norms * fix cmakelist * pyhdf in ci * bug fix obsvar name * bugfix for variable name + testoutput Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: Maryam Abdi <maryamao@ucar.edu> * Add geometric height options for geovals (#1024) * A fix for SYNOP data containing BUFR subsets (#1021) * interim code, not yet debugged/working * handling subsets, is it even needed * placement of data into meta_data * much more refactoring, taking out repfacs that make zero sense in synop data * after more bug fixes to SYNOP converter * after more bug fixes to SYNOP converter * revert back reference file that uses bufr2ioda.x * bug fix for virtual temp and surface pres * fixing coding norms Co-authored-by: gthompsnJCSDA <gthompsn@ucar.edu> * Fix combine obsspace bug (#1027) Co-authored-by: Patrick Nichols <patjnichols@gmail.com> * Updated with ioda engines header file name change (Factory.h -> EngineUtils.h) (#1030) * Modified tests for MacOS SIP compatibility. (#1011) Co-authored-by: Patrick Nichols <patjnichols@gmail.com> * Changes in L2 trace gas retrievals converters for column operator generalization (#1025) * changes in iodaconv to make a better ufo L2 nadir operator * fix typo * change index * beging work on amf, to finish * move the amf calculation in ioda conv * update tests * fix coding norms * fix exec speed * remove amf variables and code clarity * update testinput * faster tropoomi converter execution * fix type in pressure * update test output * fix coding norms Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> * Added satbias converter for gmi_gpm. (#1042) * add ompslpnc for geos adas along with cleanup. (#1043) * add ompslpnc for geos adas along with cleanup. * fix coding norm. * tropomi better flags handling (#1037) * use better flags * coding norms * combine two bool arrays * Update src/compo/tropomi_no2_nc2ioda.py Co-authored-by: Fabio L R Diniz <45880035+fabiolrdiniz@users.noreply.github.com> * Update src/compo/tropomi_no2_nc2ioda.py Co-authored-by: Fabio L R Diniz <45880035+fabiolrdiniz@users.noreply.github.com> * Update src/compo/tropomi_no2_nc2ioda.py Co-authored-by: Fabio L R Diniz <45880035+fabiolrdiniz@users.noreply.github.com> Co-authored-by: Fabio L R Diniz <45880035+fabiolrdiniz@users.noreply.github.com> Co-authored-by: Patrick Nichols <patjnichols@gmail.com> * Add variable for GNSS bending angle data, save orginal GSI QC flags a… (#1038) * Add variable for GNSS bending angle data, save orginal GSI QC flags and other corrections, and remove VarMetaData. * Removed the change of 'GsiEffectiveQC'. * Trigger a test. * Fixed coding norms. * Fixed coding norms. Again Co-authored-by: Dan Holdaway <27729500+danholdaway@users.noreply.github.com> * Replaced "land_type_index" by "land_type_index_NPOESS" (#921) * Replaced "land_type_index" by "land_type_index_NPOESS" in "gsi_ncdiag.py". * Updated gsi_ncdiag.py. * Updated gsi_ncdiag.py. * Removed Forecast_adjusted and Forecast_unadjusted for GSI radiance outputs. * Moved some radiance variables into the new group VarMetaData. * Add variable for GNSS bending angle data, save orginal GSI QC flags and other corrections, and remove VarMetaData. * Only changed land_type_index to land_type_index_NPOESS. Co-authored-by: Patrick Nichols <patjnichols@gmail.com> * add capability to create MOPITT IODA files for DA window (#1039) * save * add test and change converter * Update src/compo/mopitt_co_nc2ioda.py Co-authored-by: Fabio L R Diniz <45880035+fabiolrdiniz@users.noreply.github.com> * Update src/compo/mopitt_co_nc2ioda.py Co-authored-by: Fabio L R Diniz <45880035+fabiolrdiniz@users.noreply.github.com> * Update src/compo/mopitt_co_nc2ioda.py Co-authored-by: Fabio L R Diniz <45880035+fabiolrdiniz@users.noreply.github.com> * change to time time range Co-authored-by: Fabio L R Diniz <45880035+fabiolrdiniz@users.noreply.github.com> Co-authored-by: Patrick Nichols <patjnichols@gmail.com> * Feature/query cxx (#1006) * First cut at calling the new nceplibs bufr query interface. * Updated to handle multi-dimensional data. Fixed DatetimeVariable. Got bufr mhs example to work. * Added support for CCITT IA5 string fields. Fixed many other fields that not not properly support queries. * Added support for forfield. Cleaned up some uneccesarrily complicated data structures. * Removed split map in favor of a list of splits to make code more readable. * Fixed bug with transforms. Updated some YAML files. * Updated the README file. * Fixed bug were CCITT IA5 fields would fail because of a silly mistake (throwing exception unneccessarliy). * Renamed for field to group_by field. * Removed duplicate directory for transforms. * incremental checkin, getting higher dimensional data working * Added DataObject cpp * Finished major code revisions and got things to compile for automatic dimensioning. * Some bug fixes to get simple radiance data working. * Enhanced the reading of th dimensions field. * Got filtering and splitting to work again. Might need to be a little more careful with slice dimensions, but works with existing examples. * Fixed a dimension naming issue with the way group_by location dimensions were being named. Fixed some compiler warnings. * Added arguments to bufr2ioda so you can limit the amount of messages that are parsed. * Changed dimensions attribute to list or single string. * Fixed transforms. * Fixed transfoms so they will not affect missing values. * Fixed bug that happened when we tried to export an empty category. * Fixed strange issue that happened while applying filters. Removed some redundant yaml files from testinput. * Updated a bunch of unit tests so that they would work with the latest code. * updated test output files to work with latest version of ioda * Added unit test and now time the runtime. * Fixed a couple more unit tests. * Added .dump files to gitlfs * removed uneeded file * BUFR2IODA conversion for AMSU-A and MHS Radiance data (#713) * Add bufr to ioda conversion for IASI, ERAS/DB AMSU-A and MHS for ncar-bufr2nc-fortarn utility * Add tests and associated test input/output files for ncar-bufr2nc-fortarn utility * Add NCEP BUFR tables for normal-feed and RARS/DB satellite radiance data * Add YAML files, tests, and test reference for normal-feed and db/ears AMSU-A * Fix coding norm * Rename NCEP BUFR tables for satellite radiance data * Add yaml files, tests, and test input/output files for MHS (normal-feed and RARS/DB) t * Update Split.cpp * Update DataObject.cpp * Update IodaEncoder.cpp * Update bufr_ncep_1bamua.yaml * Update bufr_ncep_1bmhs.yaml * Update bufr_ncep_esamua.yaml * Update bufr_ncep_esmhs.yaml * Fixed coding norms issue. * Fixed some little issues. Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Update README.md Updated documentation for dimensions. * Test and enhance two yaml files for adpsfc snow and sevcrs data (#736) * Update snowadpsfc * Update bufr_snow_adpsfc.yaml * Update testoutput * Update yaml files and testinput and testoutput data * Reduce input data size to 20% and update output * Reduce input data size to 250KB and update testoutput * Reduce the size of test files for amsua, mhs and iasi. Update the corresponding test output files (#764) * Removed uneeded files I missed during merge. * Initial checkin of query classes with a lot of placeholder code * Got data collection mostly working. * Starting to work end to end. Need to fix exportdims and revisit code that deals with group by fields. * Fixed issue where targets and masks were being copied unecessarily. * Feature/adpupa prepbufr (#794) * Added adpupa prepbufr yamls w/o group_by field with input adpupa prepbufr file * adpupa netCDF files w/o group_by field * Updates made after ioda-validate.x run. * Changed Quality Mark group from "ObsQualityMark" to "QualityMarker" * Got a bunch of unit tests working. * add aircft profiles to feature/query * Fixed non-working functional unit tests. * Added unit test for simple groupby example. * Made simple groupby test work. Fixed related bugs. * Added unit test for bufr files with missing fields. * Added unit test for sfcshp data * accidentaly messed up the bufr-mhs test. this fixes it * Integrated latest bug fixes from the feature/query branch. * Fixed bug were we weren't checking for empty sequence counts. * added adpupa tests * Fixed bugs that prevented adpupa from working. * Getting much closer to having adpupa groupby example working. * fixed final bug to get adpupa groupby case to work the same way. * Removed some uneeded code. * Reduce the size of the ADPUPA test input furrther * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. Fixed some copyright statements. * Changed the way data provider works (no longer a singleton). Minor cleanup. * Added print_queries tool to the ioda-converter project. * Improved situation arrising from missing string fields. * Improved detection of string fields. * Moved table logic out of the print_quiries function. Fixed bug were idxs were not being displayed on queries. * Replace nlocs with Location, reduce the size of ADPUPA.prepbufr and … (#884) * Replace nlocs with Location. Reduce the size of ADPUPA.prepbufr and update the test output. * remove checking of satellite id in QC (#878) * remove checking of satellite id in QC * added checking of bits 1 and 4 in QC flag Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> * Feature/optional python qc (#881) * Corrected bit checks due to reverse ordered bits * add check if there any data to write out * update ctest input and reference file * add new test output verification * corrected pycodestyle error * added optional georeality QC checks similar to those present in FORTRAN converter * pycodestyle corrections * removed percent confidence check * attempting fix of conflicts with existing code * corrected syntax errors * removed global variable Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: haydenlj <99764299+haydenlj@users.noreply.github.com> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Reverted ADPUPA bufr file to smaller one. * Fixed compiler warnings * Fixed code style (norm) issues. * Added doc strings to query code. * Feature/query cxx praveen (#893) * adpupa prepbufr yaml for updated group_by field and corresponding testoutput file * adpupa prepbufr yaml containing GSI mnemonics only and corresponding output file * adpupa gsi prepbufr file with updated name, groups, units, with bufr2ioda_cxx.x generated netCDF file * added TVO and CLTP to the adpupa prepbufr yaml, updated name and output netcdf files * Updated dimensions into yaml * Added group_by fields for ELV, TYP, and T29; and updated the nc files * Added white space * Change dimension name to following unified IODA naming convention (#767) * Reduce the size of test files for amsua, mhs and iasi. Update the corresponding test output files * Replace "nlocs" with "Location" and "nchans" with "Channel" (to following the unified IODA naming convention). * Fix a typo and update test output accordingly * Added white space * Made updates to make everything consistent with query_cxx branch Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Fixed bug when there were more than one subset in the file. Added ability to print monotone output (so you can pipe to a text file). * Updated print queries so it no longer uses colors but prints the dimensioning subpaths seperatly. * Chenged some maps to unordered_maps as they are better performance wise. * Feature/query cxx as (#914) * Updated yaml, yamlinput, and yaml output for SatWind AVHRR NC005080 within the new feature/query_cxx branch. * The bufr_ncep_sawtind_avhrr yaml has been updated to include and produce input and output with the same name scheme. Files have been renamed accordingly. * Modified input and output fields of the yaml file. * modified: CMakeLists.txt renamed and modified: bufr_ncep_satwind_avhrr.yaml -> testinput/bufr_ncep_satwind_avhrr.yaml The input fields and linked name fields for "time" has been updated as suggested. The CMakeList has been updated to match with the new naming scheme. The "dimensions" field has been updated as suggested. And the yaml has now been moved from ./test to ./test/testinput. * Removed indices for most variables, while adding in 2 more indices for GNAP and PCCF to get wind direction and wind speed generating application and percent confidence. * Updated variables names to match the JEDI convention spreadsheet. * Removed a default/backup configuration line. * Modify unit attribute in BUFR2IODA converter and bug fix for satellite winds (#940) * Make units attribute an optional variable * Bug fix for AVHRR satellite wind test: (1) unit in yaml (2) output directory in yaml (3) test output file in CMakelist.txt * Change the test input and output file names according to the following logic: (1) for input, use the global dump file naming convention (e.g. gdas.t00z.satwnd.tm00.bufr_d) (2) for output change the input file name extension bufr_d to nc (gdas.t00z.satwnd.tm00.nc) * delete bufr_ncep_satwind_avhrr.nc * Make sure the time info is consistent between the file name and the data in the file. * adjust the order of files listed in the CMakeList.txt * Change the way to test the empty string. * Fixed issue where non-existing query with index would cause a crash. Also fixed bug where the suggested dimesioning paths would be messed up sometimes. * Fixed some norm test failures. * Fixed all the unit tests * Made change so that there is always one group_by field whcih applies to all the variables. * Removed uneeded test * Improved some of the code to do with finding the data object to group against. * Fixed bug in print queries were fixed repeats were being missed. * update two yaml files via query_cxx (#906) * update snow adpsfc yaml and the result * Update sevcsr yaml file and the result * Update bufr_ncep_read_2_dim_blocks.yaml * Fix dateTim with DateTime, nchans with Channel * Remove ncep for yaml and modify the CMakeLists.txt * Update bufr_ncep_snow_adpsfc.yaml * Update bufr_read_2_dim_blocks.yaml to follow JEDI convention and update the associated test output (bufr_read_2_dim_blocks.nc). * Modified yaml, add sacle and update the result * Changes cloudAmount unit and update the result * Change cloudAmount to cloudCoverTotal * Set BUFR variable: CLDMNT to cloudAmount since it is already in JEDI convention with optional dimenion as [location, channel] Co-authored-by: Emily Liu <emily.liu@noaa.gov> * Feature/query cxx mhs (#960) * Update 1bmhs YAML and test output * Update DB mhs YAML file and test output * Modify longnames and update test output files * Feature/query cxx amsua (#958) * Update YAMLS and test output files for 1bamsua and esamsua * Update YAML files for AMSU-A * Remove unwanted test file for amsua-a * fix spacing. * add spacings between lines * Change height to heightOfStation and update test output * Fix longnames * Feature/query cxx iasi (#966) * Update IASI yaml and related testoutput * Update IASI YAML * Update IASI test file * Fix a few errors in IASI YAML and update the test output * Feature/query cxx nick e adpsfc (#904) * adpsfc prepbufr and bufr * recommended changes * recommended updates 2 (* and deg) * CMakelists.txt update * update CMakeLists.txt * adpsfc prepbufr recommended changes * remove nc, add full adpsfcyaml * adpsfc updates, plus CMakelists * remove CLTP and VSSO * decrease number of variables * update output * heightOfStation -> stationElevation * get rid of oceanographic * update heightofbaseofcloud description * update with correct files * updates to HOCB CLAM VSSO * update thinned, has all subsets * Fixed the unit tests * reportid -> SID, CLAM[1] is CLAM, pW@Obs, percent ->1 * update output * new output from updated bufr2ioda.x * bufr: HSMSL+SELV=stationElevation, PB: percent->1 * get rid of units:"" and unitless. * add dimensions. cloudAmountObserved. cloudCover/100 * moistureQM -> dewpointTemperatureQM * cloudAmount no units * updated bufr with new dim names * update prep. few names. add dimens * most updates for prep * No more 'Seen'. Other updates * delete metarsequence, Of not of * Move VSSO to be with other variables related to clouds * Rearrange the order of entries in the YAML * fix spacing in the YAML * Reorder entries in YAML * fix spaceing * Rearrange entries * Modify variable names and bug fix for ADPSFC BUFR and PREPBUFR YAMLS. Test output files also updated. * Update CMakelist.txt and add comments * Remove obselete files * Modify ADPSFC test output back to original * Modify variable names for max/min Temperatures * Add missing variables and rearranged entries. * Update test output for ADPSFC * Change heightOfStation to heightOfObservation * Update ctest output file for prepbufr adpsfc * get rid of v1 adpsfc. final recommended updates * revert, keep gdas.t06z.adpsfc.tm00.bufr_d for adpsfc_snow * re-add adpsfc_snow input bufr to CmakeLists.txt * Add the correct input test file for adpsfc snow data * update prepbufr file names, CMake, yaml obsdatain and out * typo * degrees_north and east -> degree_north and east * delete 'adsfc' file, update with degree Co-authored-by: Emily Liu <emily.liu@noaa.gov> Co-authored-by: Ron McLaren <ronlaren@aol.com> * Feature/query cxx atovs (#963) * Update test input files so data from all related satellite platform are included. Rearrange the entries in the YAML files. * update tes output files so that the naming convention of the filename is consistent. * Update test output for iodaconv fortran * Updated DatetimeVariable.cpp code for optional value of Minute (#949) * Updated DatetimeVariable.cpp code for optional value of Minute * added a yaml file to test the DatetimeVariable.cpp code for optional MINU variable Co-authored-by: Emily Liu <emily.liu@noaa.gov> * Now uses OOPS logging. * Fixed some comments * cast time to seconds instead of dividing by 1000 * fixed norm test failure * made exit into exception * using back inserter to append to sequence path * fixed error message. * fixed error message. * added prepbufr to git lfs * removed line continuation characters. * replaced 10e10 with missing value * accidentally deleted line of code. added it back * changed print time function back tot he original. * using gsl::narrow instead of simpoe static cast * Enhanced vector product function * Pulled out missing value into a Constants.h file to avoid duplication. * fixed some comments * fixed file permissions * Added try catch statements to main functions. Fixed warning. * pre reserve some memory * removed unused test from CMakeLists.txt file. * Added include of unordered map as some compilers were not finding it. * Reverted unintended changes to files in ncar-bufr2nc-fortran and lib-python * Fixed bug that invovled parsing the bufr table data incorrectly. Basically we were not exiting trivial sequences properly when parsing the table. * Fixed bug in print queries as it was not trimming subset trims properly and was getting confused. * Fixed some warnings and errors that happen with the CLANG compiler. * Enable the C++14 standard for C++ compiles. (#1036) * Fixed some code review issues that Cory Martin broght up. * updated docs for DataObject::print function * commented in line that was accidentally commented out * Revert "commented in line that was accidentally commented out" This reverts commit c78b93eae745b64b45567cec0ec2388ec129d0fb. * Added back missing tests from the develop branch. * Moved test_iodaconv_generic_gnssro_bufr back to where it was before (positionaly). * Fixed issue caused by code review comment fix that caused bufr2ioda.x to obfuscate excepotion descriptions. * Fixed exception handler in print_queries.x. * Fixed Mac SIP issues. * Commented out Emily Liu's bufr2nc_fortran.x tests.. * trigger ci * changes needed for new containers (#1056) * changes needed for new containers * debug * debug * debug * First stab at fixing CI using new containers * Fix jedi-cmake * Try make instead of ctest * Try ctest after make * Try ctest -j4 for intel * Try ctest -j8 for intel * Remove comments from buildspecs * More cleanup in CI/buildspec_gnu.yml * Fix CDASH links Co-authored-by: Dom Heinzeller <dom.heinzeller@icloud.com> Co-authored-by: Emily Liu <emily.liu@noaa.gov> Co-authored-by: emilyhcliu <36091766+emilyhcliu@users.noreply.github.com> Co-authored-by: YoulongXia-NOAA <51668688+YoulongXia-NOAA@users.noreply.github.com> Co-authored-by: PraveenKumar-NOAA <63739712+PraveenKumar-NOAA@users.noreply.github.com> Co-authored-by: Nicholas.Esposito <Nicholas.Esposito@noaa.gov> Co-authored-by: haydenlj <99764299+haydenlj@users.noreply.github.com> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: AshleyStanfield-NOAA <92459593+AshleyStanfield-NOAA@users.noreply.github.com> Co-authored-by: NicholasEsposito-NOAA <62616739+nicholasesposito@users.noreply.github.com> Co-authored-by: Stephen Herbener <stephen.herbener@gmail.com> Co-authored-by: Cory Martin <cory.r.martin@noaa.gov> Co-authored-by: Patrick Nichols <patjnichols@gmail.com> Co-authored-by: Maryam Abdi-Oskouei <maryamao@ucar.edu> Co-authored-by: Dom Heinzeller <dom.heinzeller@icloud.com> * change from two composition related ctest to nonzero tolerances (#1064) * bugfix (#1065) Co-authored-by: Dom Heinzeller <dom.heinzeller@icloud.com> * fix typo index (#1057) Co-authored-by: Maryam Abdi <maryamao@ucar.edu> Co-authored-by: Dom Heinzeller <dom.heinzeller@icloud.com> * Feature/query dim naming bug (#1040) * First cut at calling the new nceplibs bufr query interface. * Updated to handle multi-dimensional data. Fixed DatetimeVariable. Got bufr mhs example to work. * Added support for CCITT IA5 string fields. Fixed many other fields that not not properly support queries. * Added support for forfield. Cleaned up some uneccesarrily complicated data structures. * Removed split map in favor of a list of splits to make code more readable. * Fixed bug with transforms. Updated some YAML files. * Updated the README file. * Fixed bug were CCITT IA5 fields would fail because of a silly mistake (throwing exception unneccessarliy). * Renamed for field to group_by field. * Removed duplicate directory for transforms. * incremental checkin, getting higher dimensional data working * Added DataObject cpp * Finished major code revisions and got things to compile for automatic dimensioning. * Some bug fixes to get simple radiance data working. * Enhanced the reading of th dimensions field. * Got filtering and splitting to work again. Might need to be a little more careful with slice dimensions, but works with existing examples. * Fixed a dimension naming issue with the way group_by location dimensions were being named. Fixed some compiler warnings. * Added arguments to bufr2ioda so you can limit the amount of messages that are parsed. * Changed dimensions attribute to list or single string. * Fixed transforms. * Fixed transfoms so they will not affect missing values. * Fixed bug that happened when we tried to export an empty category. * Fixed strange issue that happened while applying filters. Removed some redundant yaml files from testinput. * Updated a bunch of unit tests so that they would work with the latest code. * updated test output files to work with latest version of ioda * Added unit test and now time the runtime. * Fixed a couple more unit tests. * Added .dump files to gitlfs * removed uneeded file * BUFR2IODA conversion for AMSU-A and MHS Radiance data (#713) * Add bufr to ioda conversion for IASI, ERAS/DB AMSU-A and MHS for ncar-bufr2nc-fortarn utility * Add tests and associated test input/output files for ncar-bufr2nc-fortarn utility * Add NCEP BUFR tables for normal-feed and RARS/DB satellite radiance data * Add YAML files, tests, and test reference for normal-feed and db/ears AMSU-A * Fix coding norm * Rename NCEP BUFR tables for satellite radiance data * Add yaml files, tests, and test input/output files for MHS (normal-feed and RARS/DB) t * Update Split.cpp * Update DataObject.cpp * Update IodaEncoder.cpp * Update bufr_ncep_1bamua.yaml * Update bufr_ncep_1bmhs.yaml * Update bufr_ncep_esamua.yaml * Update bufr_ncep_esmhs.yaml * Fixed coding norms issue. * Fixed some little issues. Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Update README.md Updated documentation for dimensions. * Test and enhance two yaml files for adpsfc snow and sevcrs data (#736) * Update snowadpsfc * Update bufr_snow_adpsfc.yaml * Update testoutput * Update yaml files and testinput and testoutput data * Reduce input data size to 20% and update output * Reduce input data size to 250KB and update testoutput * Reduce the size of test files for amsua, mhs and iasi. Update the corresponding test output files (#764) * Removed uneeded files I missed during merge. * Initial checkin of query classes with a lot of placeholder code * Got data collection mostly working. * Starting to work end to end. Need to fix exportdims and revisit code that deals with group by fields. * Fixed issue where targets and masks were being copied unecessarily. * Feature/adpupa prepbufr (#794) * Added adpupa prepbufr yamls w/o group_by field with input adpupa prepbufr file * adpupa netCDF files w/o group_by field * Updates made after ioda-validate.x run. * Changed Quality Mark group from "ObsQualityMark" to "QualityMarker" * Got a bunch of unit tests working. * add aircft profiles to feature/query * Fixed non-working functional unit tests. * Added unit test for simple groupby example. * Made simple groupby test work. Fixed related bugs. * Added unit test for bufr files with missing fields. * Added unit test for sfcshp data * accidentaly messed up the bufr-mhs test. this fixes it * Integrated latest bug fixes from the feature/query branch. * Fixed bug were we weren't checking for empty sequence counts. * added adpupa tests * Fixed bugs that prevented adpupa from working. * Getting much closer to having adpupa groupby example working. * fixed final bug to get adpupa groupby case to work the same way. * Removed some uneeded code. * Reduce the size of the ADPUPA test input furrther * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. Fixed some copyright statements. * Changed the way data provider works (no longer a singleton). Minor cleanup. * Added print_queries tool to the ioda-converter project. * Improved situation arrising from missing string fields. * Improved detection of string fields. * Moved table logic out of the print_quiries function. Fixed bug were idxs were not being displayed on queries. * Replace nlocs with Location, reduce the size of ADPUPA.prepbufr and … (#884) * Replace nlocs with Location. Reduce the size of ADPUPA.prepbufr and update the test output. * remove checking of satellite id in QC (#878) * remove checking of satellite id in QC * added checking of bits 1 and 4 in QC flag Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> * Feature/optional python qc (#881) * Corrected bit checks due to reverse ordered bits * add check if there any data to write out * update ctest input and reference file * add new test output verification * corrected pycodestyle error * added optional georeality QC checks similar to those present in FORTRAN converter * pycodestyle corrections * removed percent confidence check * attempting fix of conflicts with existing code * corrected syntax errors * removed global variable Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: haydenlj <99764299+haydenlj@users.noreply.github.com> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Reverted ADPUPA bufr file to smaller one. * Fixed compiler warnings * Fixed code style (norm) issues. * Added doc strings to query code. * Feature/query cxx praveen (#893) * adpupa prepbufr yaml for updated group_by field and corresponding testoutput file * adpupa prepbufr yaml containing GSI mnemonics only and corresponding output file * adpupa gsi prepbufr file with updated name, groups, units, with bufr2ioda_cxx.x generated netCDF file * added TVO and CLTP to the adpupa prepbufr yaml, updated name and output netcdf files * Updated dimensions into yaml * Added group_by fields for ELV, TYP, and T29; and updated the nc files * Added white space * Change dimension name to following unified IODA naming convention (#767) * Reduce the size of test files for amsua, mhs and iasi. Update the corresponding test output files * Replace "nlocs" with "Location" and "nchans" with "Channel" (to following the unified IODA naming convention). * Fix a typo and update test output accordingly * Added white space * Made updates to make everything consistent with query_cxx branch Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Fixed bug when there were more than one subset in the file. Added ability to print monotone output (so you can pipe to a text file). * Updated print queries so it no longer uses colors but prints the dimensioning subpaths seperatly. * Chenged some maps to unordered_maps as they are better performance wise. * Feature/query cxx as (#914) * Updated yaml, yamlinput, and yaml output for SatWind AVHRR NC005080 within the new feature/query_cxx branch. * The bufr_ncep_sawtind_avhrr yaml has been updated to include and produce input and output with the same name scheme. Files have been renamed accordingly. * Modified input and output fields of the yaml file. * modified: CMakeLists.txt renamed and modified: bufr_ncep_satwind_avhrr.yaml -> testinput/bufr_ncep_satwind_avhrr.yaml The input fields and linked name fields for "time" has been updated as suggested. The CMakeList has been updated to match with the new naming scheme. The "dimensions" field has been updated as suggested. And the yaml has now been moved from ./test to ./test/testinput. * Removed indices for most variables, while adding in 2 more indices for GNAP and PCCF to get wind direction and wind speed generating application and percent confidence. * Updated variables names to match the JEDI convention spreadsheet. * Removed a default/backup configuration line. * Modify unit attribute in BUFR2IODA converter and bug fix for satellite winds (#940) * Make units attribute an optional variable * Bug fix for AVHRR satellite wind test: (1) unit in yaml (2) output directory in yaml (3) test output file in CMakelist.txt * Change the test input and output file names according to the following logic: (1) for input, use the global dump file naming convention (e.g. gdas.t00z.satwnd.tm00.bufr_d) (2) for output change the input file name extension bufr_d to nc (gdas.t00z.satwnd.tm00.nc) * delete bufr_ncep_satwind_avhrr.nc * Make sure the time info is consistent between the file name and the data in the file. * adjust the order of files listed in the CMakeList.txt * Change the way to test the empty string. * Fixed issue where non-existing query with index would cause a crash. Also fixed bug where the suggested dimesioning paths would be messed up sometimes. * Fixed some norm test failures. * Fixed all the unit tests * Made change so that there is always one group_by field whcih applies to all the variables. * Removed uneeded test * Improved some of the code to do with finding the data object to group against. * Fixed bug in print queries were fixed repeats were being missed. * update two yaml files via query_cxx (#906) * update snow adpsfc yaml and the result * Update sevcsr yaml file and the result * Update bufr_ncep_read_2_dim_blocks.yaml * Fix dateTim with DateTime, nchans with Channel * Remove ncep for yaml and modify the CMakeLists.txt * Update bufr_ncep_snow_adpsfc.yaml * Update bufr_read_2_dim_blocks.yaml to follow JEDI convention and update the associated test output (bufr_read_2_dim_blocks.nc). * Modified yaml, add sacle and update the result * Changes cloudAmount unit and update the result * Change cloudAmount to cloudCoverTotal * Set BUFR variable: CLDMNT to cloudAmount since it is already in JEDI convention with optional dimenion as [location, channel] Co-authored-by: Emily Liu <emily.liu@noaa.gov> * Feature/query cxx mhs (#960) * Update 1bmhs YAML and test output * Update DB mhs YAML file and test output * Modify longnames and update test output files * Feature/query cxx amsua (#958) * Update YAMLS and test output files for 1bamsua and esamsua * Update YAML files for AMSU-A * Remove unwanted test file for amsua-a * fix spacing. * add spacings between lines * Change height to heightOfStation and update test output * Fix longnames * Feature/query cxx iasi (#966) * Update IASI yaml and related testoutput * Update IASI YAML * Update IASI test file * Fix a few errors in IASI YAML and update the test output * Feature/query cxx nick e adpsfc (#904) * adpsfc prepbufr and bufr * recommended changes * recommended updates 2 (* and deg) * CMakelists.txt update * update CMakeLists.txt * adpsfc prepbufr recommended changes * remove nc, add full adpsfcyaml * adpsfc updates, plus CMakelists * remove CLTP and VSSO * decrease number of variables * update output * heightOfStation -> stationElevation * get rid of oceanographic * update heightofbaseofcloud description * update with correct files * updates to HOCB CLAM VSSO * update thinned, has all subsets * Fixed the unit tests * reportid -> SID, CLAM[1] is CLAM, pW@Obs, percent ->1 * update output * new output from updated bufr2ioda.x * bufr: HSMSL+SELV=stationElevation, PB: percent->1 * get rid of units:"" and unitless. * add dimensions. cloudAmountObserved. cloudCover/100 * moistureQM -> dewpointTemperatureQM * cloudAmount no units * updated bufr with new dim names * update prep. few names. add dimens * most updates for prep * No more 'Seen'. Other updates * delete metarsequence, Of not of * Move VSSO to be with other variables related to clouds * Rearrange the order of entries in the YAML * fix spacing in the YAML * Reorder entries in YAML * fix spaceing * Rearrange entries * Modify variable names and bug fix for ADPSFC BUFR and PREPBUFR YAMLS. Test output files also updated. * Update CMakelist.txt and add comments * Remove obselete files * Modify ADPSFC test output back to original * Modify variable names for max/min Temperatures * Add missing variables and rearranged entries. * Update test output for ADPSFC * Change heightOfStation to heightOfObservation * Update ctest output file for prepbufr adpsfc * get rid of v1 adpsfc. final recommended updates * revert, keep gdas.t06z.adpsfc.tm00.bufr_d for adpsfc_snow * re-add adpsfc_snow input bufr to CmakeLists.txt * Add the correct input test file for adpsfc snow data * update prepbufr file names, CMake, yaml obsdatain and out * typo * degrees_north and east -> degree_north and east * delete 'adsfc' file, update with degree Co-authored-by: Emily Liu <emily.liu@noaa.gov> Co-authored-by: Ron McLaren <ronlaren@aol.com> * Feature/query cxx atovs (#963) * Update test input files so data from all related satellite platform are included. Rearrange the entries in the YAML files. * update tes output files so that the naming convention of the filename is consistent. * Update test output for iodaconv fortran * Updated DatetimeVariable.cpp code for optional value of Minute (#949) * Updated DatetimeVariable.cpp code for optional value of Minute * added a yaml file to test the DatetimeVariable.cpp code for optional MINU variable Co-authored-by: Emily Liu <emily.liu@noaa.gov> * Now uses OOPS logging. * Fixed some comments * cast time to seconds instead of dividing by 1000 * fixed norm test failure * made exit into exception * using back inserter to append to sequence path * fixed error message. * fixed error message. * added prepbufr to git lfs * removed line continuation characters. * replaced 10e10 with missing value * accidentally deleted line of code. added it back * changed print time function back tot he original. * using gsl::narrow instead of simpoe static cast * Enhanced vector product function * Pulled out missing value into a Constants.h file to avoid duplication. * fixed some comments * fixed file permissions * Added try catch statements to main functions. Fixed warning. * pre reserve some memory * removed unused test from CMakeLists.txt file. * Added include of unordered map as some compilers were not finding it. * Reverted unintended changes to files in ncar-bufr2nc-fortran and lib-python * Fixed bug that invovled parsing the bufr table data incorrectly. Basically we were not exiting trivial sequences properly when parsing the table. * Fixed bug in print queries as it was not trimming subset trims properly and was getting confused. * Fixed some warnings and errors that happen with the CLANG compiler. * Enable the C++14 standard for C++ compiles. (#1036) * Fixed some code review issues that Cory Martin broght up. * updated docs for DataObject::print function * commented in line that was accidentally commented out * Revert "commented in line that was accidentally commented out" This reverts commit c78b93eae745b64b45567cec0ec2388ec129d0fb. * Added back missing tests from the develop branch. * Moved test_iodaconv_generic_gnssro_bufr back to where it was before (positionaly). * Forcing named dimensions to be generic paths (not based on subset). Basically they must start with *. * Fixed issue caused by code review comment fix that caused bufr2ioda.x to obfuscate excepotion descriptions. * Fixed exception handler in print_queries.x. Co-authored-by: Emily Liu <emily.liu@noaa.gov> Co-authored-by: emilyhcliu <36091766+emilyhcliu@users.noreply.github.com> Co-authored-by: YoulongXia-NOAA <51668688+YoulongXia-NOAA@users.noreply.github.com> Co-authored-by: PraveenKumar-NOAA <63739712+PraveenKumar-NOAA@users.noreply.github.com> Co-authored-by: Nicholas.Esposito <Nicholas.Esposito@noaa.gov> Co-authored-by: haydenlj <99764299+haydenlj@users.noreply.github.com> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: AshleyStanfield-NOAA <92459593+AshleyStanfield-NOAA@users.noreply.github.com> Co-authored-by: NicholasEsposito-NOAA <62616739+nicholasesposito@users.noreply.github.com> Co-authored-by: Stephen Herbener <stephen.herbener@gmail.com> Co-authored-by: Cory Martin <cory.r.martin@noaa.gov> Co-authored-by: Patrick Nichols <patjnichols@gmail.com> * ioda converter for synthetic SWOT observations (#1058) * Adding SWOT ADT converter * adding swot l2 ssh 2 adt ioda converter and test data * adding the data to CMakelists * Reducing test file and updating adt estimate * fix code norms Co-authored-by: Kriti <> Co-authored-by: Travis Sluka <travissluka@gmail.com> Co-authored-by: Benjamin Ruston <93737224+BenjaminRuston@users.noreply.github.com> * Fix reference to oz_sensors in proc_gsi_ncdiag (#1053) * Fix oz_sensor issue * geoval fix too * empty commit to trigger CI as ufo-data was out of sync Co-authored-by: Benjamin Ruston <93737224+BenjaminRuston@users.noreply.github.com> Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: Fabio L R Diniz <45880035+fabiolrdiniz@users.noreply.github.com> * Add ability to convert Text radiosonde data (TEMP format) to IODA (#1060) * initial checkin of code for converter for text (TAC) radiosonde to ioda * fixed many pieces including proper output of IODA file * adding in new share subdir and the station file for radiosondes, json format * add in small raob_stations file for rapid ctest * addition of testoutput ref file * add ctest for sonde tac2ioda and also add share mkdir and linking * fixing coding norms * add the sonde_tac2ioda.py file to CMake * need to add another utils file * fix path for since it needs to be relative to test * another update to relative path for share * forgot the testinput small file * fix testoutput filename for linking * update ref file with changed attribute of source file Co-authored-by: gthompsnJCSDA <gthompsn@ucar.edu> Co-authored-by: Benjamin Ruston <93737224+BenjaminRuston@users.noreply.github.com> * Updated viirs_aod2ioda.py to include nchans for channel number (#1046) * Updated viirs_aod2ioda.py to include nchans for channel number * Fixed coding norms Co-authored-by: Patrick Nichols <patjnichols@gmail.com> Co-authored-by: Benjamin Ruston <93737224+BenjaminRuston@users.noreply.github.com> * Feature/query cxx types (#1044) * First cut at calling the new nceplibs bufr query interface. * Updated to handle multi-dimensional data. Fixed DatetimeVariable. Got bufr mhs example to work. * Added support for CCITT IA5 string fields. Fixed many other fields that not not properly support queries. * Added support for forfield. Cleaned up some uneccesarrily complicated data structures. * Removed split map in favor of a list of splits to make code more readable. * Fixed bug with transforms. Updated some YAML files. * Updated the README file. * Fixed bug were CCITT IA5 fields would fail because of a silly mistake (throwing exception unneccessarliy). * Renamed for field to group_by field. * Removed duplicate directory for transforms. * incremental checkin, getting higher dimensional data working * Added DataObject cpp * Finished major code revisions and got things to compile for automatic dimensioning. * Some bug fixes to get simple radiance data working. * Enhanced the reading of th dimensions field. * Got filtering and splitting to work again. Might need to be a little more careful with slice dimensions, but works with existing examples. * Fixed a dimension naming issue with the way group_by location dimensions were being named. Fixed some compiler warnings. * Added arguments to bufr2ioda so you can limit the amount of messages that are parsed. * Changed dimensions attribute to list or single string. * Fixed transforms. * Fixed transfoms so they will not affect missing values. * Fixed bug that happened when we tried to export an empty category. * Fixed strange issue that happened while applying filters. Removed some redundant yaml files from testinput. * Updated a bunch of unit tests so that they would work with the latest code. * updated test output files to work with latest version of ioda * Added unit test and now time the runtime. * Fixed a couple more unit tests. * Added .dump files to gitlfs * removed uneeded file * BUFR2IODA conversion for AMSU-A and MHS Radiance data (#713) * Add bufr to ioda conversion for IASI, ERAS/DB AMSU-A and MHS for ncar-bufr2nc-fortarn utility * Add tests and associated test input/output files for ncar-bufr2nc-fortarn utility * Add NCEP BUFR tables for normal-feed and RARS/DB satellite radiance data * Add YAML files, tests, and test reference for normal-feed and db/ears AMSU-A * Fix coding norm * Rename NCEP BUFR tables for satellite radiance data * Add yaml files, tests, and test input/output files for MHS (normal-feed and RARS/DB) t * Update Split.cpp * Update DataObject.cpp * Update IodaEncoder.cpp * Update bufr_ncep_1bamua.yaml * Update bufr_ncep_1bmhs.yaml * Update bufr_ncep_esamua.yaml * Update bufr_ncep_esmhs.yaml * Fixed coding norms issue. * Fixed some little issues. Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Update README.md Updated documentation for dimensions. * Test and enhance two yaml files for adpsfc snow and sevcrs data (#736) * Update snowadpsfc * Update bufr_snow_adpsfc.yaml * Update testoutput * Update yaml files and testinput and testoutput data * Reduce input data size to 20% and update output * Reduce input data size to 250KB and update testoutput * Reduce the size of test files for amsua, mhs and iasi. Update the corresponding test output files (#764) * Removed uneeded files I missed during merge. * Initial checkin of query classes with a lot of placeholder code * Got data collection mostly working. * Starting to work end to end. Need to fix exportdims and revisit code that deals with group by fields. * Fixed issue where targets and masks were being copied unecessarily. * Feature/adpupa prepbufr (#794) * Added adpupa prepbufr yamls w/o group_by field with input adpupa prepbufr file * adpupa netCDF files w/o group_by field * Updates made after ioda-validate.x run. * Changed Quality Mark group from "ObsQualityMark" to "QualityMarker" * Got a bunch of unit tests working. * add aircft profiles to feature/query * Fixed non-working functional unit tests. * Added unit test for simple groupby example. * Made simple groupby test work. Fixed related bugs. * Added unit test for bufr files with missing fields. * Added unit test for sfcshp data * accidentaly messed up the bufr-mhs test. this fixes it * Integrated latest bug fixes from the feature/query branch. * Fixed bug were we weren't checking for empty sequence counts. * added adpupa tests * Fixed bugs that prevented adpupa from working. * Getting much closer to having adpupa groupby example working. * fixed final bug to get adpupa groupby case to work the same way. * Removed some uneeded code. * Reduce the size of the ADPUPA test input furrther * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. Fixed some copyright statements. * Changed the way data provider works (no longer a singleton). Minor cleanup. * Added print_queries tool to the ioda-converter project. * Improved situation arrising from missing string fields. * Improved detection of string fields. * Moved table logic out of the print_quiries function. Fixed bug were idxs were not being displayed on queries. * Replace nlocs with Location, reduce the size of ADPUPA.prepbufr and … (#884) * Replace nlocs with Location. Reduce the size of ADPUPA.prepbufr and update the test output. * remove checking of satellite id in QC (#878) * remove checking of satellite id in QC * added checking of bits 1 and 4 in QC flag Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> * Feature/optional python qc (#881) * Corrected bit checks due to reverse ordered bits * add check if there any data to write out * update ctest input and reference file * add new test output verification * corrected pycodestyle error * added optional georeality QC checks similar to those present in FORTRAN converter * pycodestyle corrections * removed percent confidence check * attempting fix of conflicts with existing code * corrected syntax errors * removed global variable Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: haydenlj <99764299+haydenlj@users.noreply.github.com> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Reverted ADPUPA bufr file to smaller one. * Fixed compiler warnings * Fixed code style (norm) issues. * Added doc strings to query code. * Feature/query cxx praveen (#893) * adpupa prepbufr yaml for updated group_by field and corresponding testoutput file * adpupa prepbufr yaml containing GSI mnemonics only and corresponding output file * adpupa gsi prepbufr file with updated name, groups, units, with bufr2ioda_cxx.x generated netCDF file * added TVO and CLTP to the adpupa prepbufr yaml, updated name and output netcdf files * Updated dimensions into yaml * Added group_by fields for ELV, TYP, and T29; and updated the nc files * Added white space * Change dimension name to following unified IODA naming convention (#767) * Reduce the size of test files for amsua, mhs and iasi. Update the corresponding test output files * Replace "nlocs" with "Location" and "nchans" with "Channel" (to following the unified IODA naming convention). * Fix a typo and update test output accordingly * Added white space * Made updates to make everything consistent with query_cxx branch Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Fixed bug when there were more than one subset in the file. Added ability to print monotone output (so you can pipe to a text file). * Updated print queries so it no longer uses colors but prints the dimensioning subpaths seperatly. * Chenged some maps to unordered_maps as they are better performance wise. * Feature/query cxx as (#914) * Updated yaml, yamlinput, and yaml output for SatWind AVHRR NC005080 within the new feature/query_cxx branch. * The bufr_ncep_sawtind_avhrr yaml has been updated to include and produce input and output with the same name scheme. Files have been renamed accordingly. * Modified input and output fields of the yaml file. * modified: CMakeLists.txt renamed and modified: bufr_ncep_satwind_avhrr.yaml -> testinput/bufr_ncep_satwind_avhrr.yaml The input fields and linked name fields for "time" has been updated as suggested. The CMakeList has been updated to match with the new naming scheme. The "dimensions" field has been updated as suggested. And the yaml has now been moved from ./test to ./test/testinput. * Removed indices for most variables, while adding in 2 more indices for GNAP and PCCF to get wind direction and wind speed generating application and percent confidence. * Updated variables names to match the JEDI convention spreadsheet. * Removed a default/backup configuration line. * Modify unit attribute in BUFR2IODA converter and bug fix for satellite winds (#940) * Make units attribute an optional variable * Bug fix for AVHRR satellite wind test: (1) unit in yaml (2) output directory in yaml (3) test output file in CMakelist.txt * Change the test input and output file names according to the following logic: (1) for input, use the global dump file naming convention (e.g. gdas.t00z.satwnd.tm00.bufr_d) (2) for output change the input file name extension bufr_d to nc (gdas.t00z.satwnd.tm00.nc) * delete bufr_ncep_satwind_avhrr.nc * Make sure the time info is consistent between the file name and the data in the file. * adjust the order of files listed in the CMakeList.txt * Change the way to test the empty string. * Fixed issue where non-existing query with index would cause a crash. Also fixed bug where the suggested dimesioning paths would be messed up sometimes. * Fixed some norm test failures. * Fixed all the unit tests * Made change so that there is always one group_by field whcih applies to all the variables. * Removed uneeded test * Improved some of the code to do with finding the data object to group against. * Fixed bug in print queries were fixed repeats were being missed. * update two yaml files via query_cxx (#906) * update snow adpsfc yaml and the result * Update sevcsr yaml file and the result * Update bufr_ncep_read_2_dim_blocks.yaml * Fix dateTim with DateTime, nchans with Channel * Remove ncep for yaml and modify the CMakeLists.txt * Update bufr_ncep_snow_adpsfc.yaml * Update bufr_read_2_dim_blocks.yaml to follow JEDI convention and update the associated test output (bufr_read_2_dim_blocks.nc). * Modified yaml, add sacle and update the result * Changes cloudAmount unit and update the result * Change cloudAmount to cloudCoverTotal * Set BUFR variable: CLDMNT to cloudAmount since it is already in JEDI convention with optional dimenion as [location, channel] Co-authored-by: Emily Liu <emily.liu@noaa.gov> * Feature/query cxx mhs (#960) * Update 1bmhs YAML and test output * Update DB mhs YAML file and test output * Modify longnames and update test output files * Feature/query cxx amsua (#958) * Update YAMLS and test output files for 1bamsua and esamsua * Update YAML files for AMSU-A * Remove unwanted test file for amsua-a * fix spacing. * add spacings between lines * Change height to heightOfStation and update test output * Fix longnames * Feature/query cxx iasi (#966) * Update IASI yaml and related testoutput * Update IASI YAML * Update IASI test file * Fix a few errors in IASI YAML and update the test output * Feature/query cxx nick e adpsfc (#904) * adpsfc prepbufr and bufr * recommended changes * recommended updates 2 (* and deg) * CMakelists.txt update * update CMakeLists.txt * adpsfc prepbufr recommended changes * remove nc, add full adpsfcyaml * adpsfc updates, plus CMakelists * remove CLTP and VSSO * decrease number of variables * update output * heightOfStation -> stationElevation * get rid of oceanographic * update heightofbaseofcloud description * update with correct files * updates to HOCB CLAM VSSO * update thinned, has all subsets * Fixed the unit tests * reportid -> SID, CLAM[1] is CLAM, pW@Obs, percent ->1 * update output * new output from updated bufr2ioda.x * bufr: HSMSL+SELV=stationElevation, PB: percent->1 * get rid of units:"" and unitless. * add dimensions. cloudAmountObserved. cloudCover/100 * moistureQM -> dewpointTemperatureQM * cloudAmount no units * updated bufr with new dim names * update prep. few names. add dimens * most updates for prep * No more 'Seen'. Other updates * delete metarsequence, Of not of * Move VSSO to be with other variables related to clouds * Rearrange the order of entries in the YAML * fix spacing in the YAML * Reorder entries in YAML * fix spaceing * Rearrange entries * Modify variable names and bug fix for ADPSFC BUFR and PREPBUFR YAMLS. Test output files also updated. * Update CMakelist.txt and add comments * Remove obselete files * Modify ADPSFC test output back to original * Modify variable names for max/min Temperatures * Add missing variables and rearranged entries. * Update test output for ADPSFC * Change heightOfStation to heightOfObservation * Update ctest output file for prepbufr adpsfc * get rid of v1 adpsfc. final recommended updates * revert, keep gdas.t06z.adpsfc.tm00.bufr_d for adpsfc_snow * re-add adpsfc_snow input bufr to CmakeLists.txt * Add the correct input test file for adpsfc snow data * update prepbufr file names, CMake, yaml obsdatain and out * typo * degrees_north and east -> degree_north and east * delete 'adsfc' file, update with degree Co-authored-by: Emily Liu <emily.liu@noaa.gov> Co-authored-by: Ron McLaren <ronlaren@aol.com> * Feature/query cxx atovs (#963) * Update test input files so data from all related satellite platform are included. Rearrange the entries in the YAML files. * update tes output files so that the naming convention of the filename is consistent. * Update test output for iodaconv fortran * Added code for automated type selection. * Removed check for unit type mismatch. * Fixed bug with string fields. * Missing values now depend directly on the types (always max value for the type). Removed Result struct in favor of DataObject as it was kinda redundant. * Updated DatetimeVariable.cpp code for optional value of Minute (#949) * Updated DatetimeVariable.cpp code for optional value of Minute * added a yaml file to test the DatetimeVariable.cpp code for optional MINU variable Co-authored-by: Emily Liu <emily.liu@noaa.gov> * Automatic type selection based on NCEPLIB-bufr refernece, scale and unit. * Added target.h and updated cmakelists.txt * updated mhs test file * Added way to manually override the type. * Print queries now displays type information. * Fixed issue with discovering if field is an integer. * Added some doc strings. Fixed readme. * Improved the handling of data types when doing transforms. * Fixed issue where transforms were being applied to missing value values. * Added support for the latest changes to NCEPLIB bufr * Updated calls to NCEPLIB-bufr * Now uses OOPS logging. * Changed things back to using a NCEPLIB function to get table b data instead of other API calls. * Updated for latest NCEPLIB-bufr enhancements branch. It removed get_tabb_info and added nemtbb and nemtab. * Fixed some comments * cast time to seconds instead of dividing by 1000 * fixed norm test failure * made exit into exception * using back inserter to append to sequence path * fixed error message. * fixed error message. * added prepbufr to git lfs * removed line continuation characters. * replaced 10e10 with missing value * accidentally deleted line of code. added it back * changed print time function back tot he original. * using gsl::narrow instead of simpoe static cast * Enhanced vector product function * Pulled out missing value into a Constants.h file to avoid duplication. * fixed some comments * fixed file permissions * Added try catch statements to main functions. Fixed warning. * pre reserve some memory * removed unused test from CMakeLists.txt file. * fixed norm check issues * Added include of unordered map as some compilers were not finding it. * Reverted unintended changes to files in ncar-bufr2nc-fortran and lib-python * Fixed bug that invovled parsing the bufr table data incorrectly. Basically we were not exiting trivial sequences properly when parsing the table. * Fixed bug in print queries as it was not trimming subset trims properly and was getting confused. * merged latest from query_cxx. fixed issue in print queries where it would crash trying to get type information in some instances. * Fixed some warnings and errors that happen with the CLANG compiler. * Enable the C++14 standard for C++ compiles. (#1036) * Fixed some code review issues that Cory Martin broght up. * updated docs for DataObject::print function * commented in line that was accidentally commented out * Revert "commented in line that was accidentally commented out" This reverts commit c78b93eae745b64b45567cec0ec2388ec129d0fb. * Added back missing tests from the develop branch. * Moved test_iodaconv_generic_gnssro_bufr back to where it was before (positionaly). * Update unit tests files and fixed some bugs so everything passes now. * Fixed issue caused by code review comment fix that caused bufr2ioda.x to obfuscate excepotion descriptions. * Fixed exception handler in print_queries.x. * Updated readme with a type example. * Pushing files onto git-lfs * Removed support for unsigned types as Fortran does not support them. * Updated adpsfc types (#1034) * Updated adpsfc types * Update test data files for adpsfc bufr and prepbufr Co-authored-by: Emily Liu <emily.liu@noaa.gov> Co-authored-by: Ron McLaren <ronlaren@aol.com> Co-authored-by: Emily Liu <emily.liu@noaa.gov> Co-authored-by: emilyhcliu <36091766+emilyhcliu@users.noreply.github.com> Co-authored-by: YoulongXia-NOAA <51668688+YoulongXia-NOAA@users.noreply.github.com> Co-authored-by: PraveenKumar-NOAA <63739712+PraveenKumar-NOAA@users.noreply.github.com> Co-authored-by: Nicholas.Esposito <Nicholas.Esposito@noaa.gov> Co-authored-by: haydenlj <99764299+haydenlj@users.noreply.github.com> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: AshleyStanfield-NOAA <92459593+AshleyStanfield-NOAA@users.noreply.github.com> Co-authored-by: NicholasEsposito-NOAA <62616739+nicholasesposito@users.noreply.github.com> Co-authored-by: Stephen Herbener <stephen.herbener@gmail.com> Co-authored-by: Cory Martin <cory.r.martin@noaa.gov> Co-authored-by: Benjamin Ruston <93737224+BenjaminRuston@users.noreply.github.com> * Feature/pace l1b converter (#1062) * added pace radiance converter * Update CMakeLists.txt * .. * .. * .. * modified to pass code norm * .. * .. * fix the help part * .. * .. * cleanup and changing the testoutput file Co-authored-by: Benjamin Ruston <93737224+BenjaminRuston@users.noreply.github.com> * Feature/query labeled dims (#1055) * First cut at calling the new nceplibs bufr query interface. * Updated to handle multi-dimensional data. Fixed DatetimeVariable. Got bufr mhs example to work. * Added support for CCITT IA5 string fields. Fixed many other fields that not not properly support queries. * Added support for forfield. Cleaned up some uneccesarrily complicated data structures. * Removed split map in favor of a list of splits to make code more readable. * Fixed bug with transforms. Updated some YAML files. * Updated the README file. * Fixed bug were CCITT IA5 fields would fail because of a silly mistake (throwing exception unneccessarliy). * Renamed for field to group_by field. * Removed duplicate directory for transforms. * incremental checkin, getting higher dimensional data working * Added DataObject cpp * Finished major code revisions and got things to compile for automatic dimensioning. * Some bug fixes to get simple radiance data working. * …

…entions (#1121) * update GOES converter to data conventions * updated some marine converters for dateTime and naming conventions * updated MetaData group vars to new naming convention * switch preqc from float to int * quick change of Location to 32-bit integer, not 64-bit * changing wording to albedo from reflectance factor * fix latlon Location dimension to int32 type, not int64 * fix latlon Location dimension to int32 type, not int64 * fix indent for coding norm * another coding norm fix * import fix that was added to develop is copied here also * updating few marine converters to use dateTime properly * updating a couple test reference files * updating smap land converters and test reference files for naming conventions * amsr2 converter and test reference file updated for naming conventions * more marine and land converters updated naming convention * updated omi O3 converter for naming conventions * update NSIDC ice fraction converter to naming conventions * quick fixes for coding norms * updating mopitt CO converter to naming conventions * fix MODIS AOD converter and adopt naming conventions * update MLS ozone converter to naming conventions * updating AVHRR converter to naming conventions * update 1 of 2 sig-wave-height converters to naming convention * update copernicus absoluteDynamicTopography convert to naming conventions * fix a coding norm problem * fix overindent coding norm * change ncdiff tolerance check on some (to match develop branch) * update GNSSRO python converter to naming convention * updating testoutput file for change to fillvalue * empty commit Co-authored-by: gthompsnJCSDA <gthompsn@ucar.edu> Co-authored-by: BenjaminRuston <benr@ucar.edu>

…via PR1100) (#1138) * manual update of develop (PR1100) into sprint branch * using develop version of export.cpp file * using develop version of DataProvider.h * switch launch to releaseTime * fix GNSSRO, cosmic2 for naming conventions and validate program * fix test ref file to contain releaseTime * get ADPUPA to pass validate program * update naming conventions as well as develop branch changes to gsi_ncdiag * fix LaunchTime name from GSI side * update/modernize AIRNow converter especially for naming conventions * fix units on amsr2_icec * remove unnecessary file (prepbufr_group_by) * update simple_groupby to naming conventions * take out aod_viirs_obs file, obsolete * update bufr_empty_fields for naming conventions * elim extra quality/precision/convergence vars from mls_o3 * removing extraneous vars from omi ozone to pass validation * removing extraneous vars from ompsnm ozone to pass validation * get pace radiance to pass ioda-validate * get pace oc-l2 to pass ioda-validate * get modis-aqua l2 to pass ioda-validate * take out extras from ostia * big re-do of tropomi CO and NO2 using 2D variables * fixing minor items for getting ci-tests working * fix tropomi for coding norms * including env python3 on top of couple scripts * update filter_split to naming conventions * update bufr_hrs to naming conventions * update bufr_mhs to naming conventions * Added proper PYTHONPATH setting to the airnow ctest. (#1140) * some fixes for naming conventions, but not yet working for gsi_ncdiag.py Co-authored-by: gthompsnJCSDA <gthompsn@ucar.edu> Co-authored-by: Stephen Herbener <stephen.herbener@gmail.com> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com>

…ing conventions (#1141) * fix a batch of bufr files to pass tests with naming conventions; also take away some debug prints from AircraftAltitudeVariable * fix gsi ncdiag converter for naming conventions and test ref files * fix PYTHONPATH (again) after accidental removal * try fixing compo-coding-norms * fix coding norm prob * disable tropomi ctests since not able to use until changes in UFO and ioda fortran interface is implemented Co-authored-by: gthompsnJCSDA <gthompsn@ucar.edu>

…rader is doing. (#1142)

* changes for comvention varnames * update tests * coding norms

BenjaminRuston

well at least it wasn't a year Ryan ;0p

gthompsnJCSDA

Hurray, we are here finally!

Staging branch

1a7cd53

rhoneyager added the Sprint label Mar 7, 2022

rhoneyager assigned CoryMartin-NOAA, rhoneyager and srherbener Mar 7, 2022

rhoneyager added 2 commits March 8, 2022 11:43

Retarget ioda clone to the sprint branch

59a9f1e

Merge branch 'develop' into feature/sprint-ioda-converters

b19a123

rhoneyager commented Mar 8, 2022

View reviewed changes

rhoneyager and others added 22 commits March 8, 2022 13:51

Merge branch 'develop' into feature/sprint-ioda-converters

83c521b

nlocs to locations (#825)

38a37f0

Merge branch 'develop' into feature/sprint-ioda-converters

80b45dd

Convert ISO date string to Python datetime object (#840)

5bf58ef

* Print existing datetime var * Convert ISO string to dt object * Remove extra boolean * Make error handling work Co-authored-by: Ryan Honeyager <honeyage@ucar.edu>

Merge branch 'develop' into feature/sprint-ioda-converters

8d967fe

Merge branch 'develop' into feature/sprint-ioda-converters

cc8ccca

Merge branch 'develop' into feature/sprint-ioda-converters

6a1ef48

Merge branch 'develop' into feature/sprint-ioda-converters

215e42f

Remove two land converters with grib input (#883)

cceb60e

* Removed two land converters with grib input files * Removed two land conveters from the test/CMakeLists.txt * Removed the input and output files for two land converters

Updated smap9k test reference file.

e8324ae

Removed units attribute from vegetationOpacity variable according to …

7bf40a0

…the conventions specification. (#946)

srherbener and others added 12 commits May 27, 2022 18:26

Merge branch 'develop' into feature/sprint-ioda-converters

f15fc68

Merge branch 'develop' into feature/sprint-ioda-converters

ab14da4

Needed to go with feature/sprint-ioda-converters branch code for sett…

669d6ba

…ing up fill values instead of the develop version of that code.

Merge branch 'develop' into feature/sprint-ioda-converters

4e93cfb

More cleanup from merging in develop.

3a7af98

BenjaminRuston assigned PatNichols and unassigned rhoneyager Jan 11, 2023

Jérôme Barré added 2 commits January 12, 2023 07:54

make L2 trace gas converters work and be consistent with what the upg…

ec47029

…rader is doing. (#1142)

changes for L2 trace gas varnames to follow conventions (#1145)

48eacbe

* changes for comvention varnames * update tests * coding norms

PatNichols approved these changes Jan 18, 2023

View reviewed changes

BenjaminRuston requested review from BenjaminRuston and gthompsnJCSDA January 18, 2023 15:50

BenjaminRuston approved these changes Jan 18, 2023

View reviewed changes

gthompsnJCSDA approved these changes Jan 18, 2023

View reviewed changes

PatNichols merged commit c02e1b3 into develop Jan 18, 2023

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SPRINT: IODA-converters staging branch#817

SPRINT: IODA-converters staging branch#817
PatNichols merged 62 commits intodevelopfrom
feature/sprint-ioda-converters

rhoneyager commented Mar 7, 2022 •

edited by gthompsnJCSDA

Loading

Uh oh!

rhoneyager Mar 8, 2022

Uh oh!

srherbener commented May 13, 2022

Uh oh!

YoulongXia-NOAA commented May 13, 2022

Uh oh!

YoulongXia-NOAA commented May 13, 2022

Uh oh!

srherbener commented May 13, 2022

Uh oh!

YoulongXia-NOAA commented May 13, 2022

Uh oh!

srherbener commented May 27, 2022

Uh oh!

BenjaminRuston left a comment

Uh oh!

gthompsnJCSDA left a comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

16 participants

Conversation

rhoneyager commented Mar 7, 2022 • edited by gthompsnJCSDA Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Description

Errata

Uh oh!

rhoneyager Mar 8, 2022

Choose a reason for hiding this comment

Uh oh!

srherbener commented May 13, 2022

Uh oh!

YoulongXia-NOAA commented May 13, 2022

Uh oh!

YoulongXia-NOAA commented May 13, 2022

Uh oh!

srherbener commented May 13, 2022

Uh oh!

YoulongXia-NOAA commented May 13, 2022

Uh oh!

srherbener commented May 27, 2022

Uh oh!

BenjaminRuston left a comment

Choose a reason for hiding this comment

Uh oh!

gthompsnJCSDA left a comment

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

16 participants

rhoneyager commented Mar 7, 2022 •

edited by gthompsnJCSDA

Loading