Updated viirs_aod2ioda.py to include nchans for channel number#1046
Merged
BenjaminRuston merged 6 commits intodevelopfrom Sep 27, 2022
Merged
Updated viirs_aod2ioda.py to include nchans for channel number#1046BenjaminRuston merged 6 commits intodevelopfrom
BenjaminRuston merged 6 commits intodevelopfrom
Conversation
Contributor
CoryMartin-NOAA
left a comment
CoryMartin-NOAA
left a comment
There was a problem hiding this comment.
Coding norm failures:
./viirs_aod2ioda.py:47:38: E231 missing whitespace after ','
./viirs_aod2ioda.py:47:49: W291 trailing whitespace
./viirs_aod2ioda.py:162:28: E222 multiple spaces after operator
./viirs_aod2ioda.py:162:43: W291 trailing whitespace
CoryMartin-NOAA
approved these changes
Sep 13, 2022
Contributor
|
I tried to do a ctest. But it displayed the error below. Do I need to load other modules on Hera? Thanks.
|
Contributor
Author
|
Bo, this is a python module or python path issue. I'm assuming you have the
correct module for python. For python path, find your equivalent in your
directory path to:
PYTHONPATH=$PYTHONPATH
:/scratch1/NCEPDEV/da/Andrew.Tangborn/JEDI/gdas_app_august5_2022/gdasapp/build/lib/python3.7/pyioda/
…On Tue, Sep 13, 2022 at 1:55 PM Bo Huang ***@***.***> wrote:
I tried to do a ctest. But it displayed the error below. Do I need to load
other modules on Hera? Thanks.
- "55: import netCDF4 as nc
- 55: ModuleNotFoundError: No module named 'netCDF4'"
—
Reply to this email directly, view it on GitHub
<#1046 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/ARAAHYLK4GMCDXPSUPZDBCDV6C5Z3ANCNFSM6AAAAAAQK7VGYQ>
.
You are receiving this because you were assigned.Message ID:
***@***.***>
|
jeromebarre
requested changes
Sep 14, 2022
jeromebarre
left a comment
jeromebarre
left a comment
There was a problem hiding this comment.
@andytangborn can you please create the pull request that will modify the executable? So we have a series of PR that update this. I am not seeing anything in oops or fv3-jedi maybe I am missing something here?
Just tested fv3-jedi tests on tier2 level and gfs aerosols var tests are currently passing fine. Should they break, then please link or create the PR in the other repos. So we can test.
Also wouldn't you need to also update the input obs data in fv3-jedi-data.
Thanks!
Contributor
|
@jeromebarre I'm a bit confused by your comment. Can you clarify? The existing 'develop' converter seems to be wrong, this fixes it so that files produced by it work properly in JEDI. For example, it should be a 2D variable with channel 4, not |
I guess I am confused too :) So what the converter was producing is not what is currently in fv3-jedi-data in the testinput? Does it now with this current PR? It would have been good to have this linked to an issue were the problem is reported and explained as clearly as possible in the first place. We cannot keep having PR out of the blue like this. Thanks! |
Contributor
|
Yes, sorry, the fv3-jedi tests seem to be using 'idealized' input obs files (probably produced by @mpagowski or @bhuang95) and they do not match what this converter is producing in Sorry about not creating the issue, that fell through the cracks, we've been working at a rapid pace the last week trying to get prototype cycling going and I needed a file that would work before a meeting yesterday. Do you want a 'posterior' issue created, or is explanation in this PR sufficient? |
|
No that's fine for now. Not a big deal really. I would just like to have a better practice in general. Ideally I would like to have a system that doesn't allow you to create a mergable branch without opening an issue, that's the way it is at ECMWF for example with JIRA. We need to think of a way to have this a bit more fluid for everyone. Specific to this issue we need to make sure what is put in the converters is consistent with what is in the hofx or da tests. Is this the case? I just checked and the var names in fv3-jedi-data in the viirs testinput files are |
jeromebarre
approved these changes
Sep 14, 2022
srherbener
added a commit
that referenced
this pull request
Nov 14, 2022
…ible for November code sprint (#1119) * Add ability to capture useful radiosonde QC info (#1015) * add in important QC info for sondes * newly updated testoutput file after changes * need to abort out of BUFR messages that claim to contain compressed data Co-authored-by: gthompsnJCSDA <gthompsn@ucar.edu> * Bugfix/omi ompsnm converter get files correctly at beginning of window (#1017) * fix to grab files at start of the window. * fix missing space. Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> * Add a new group in the IODA format for L2 products with averaging kernel and slant columns (#1018) * firt commit, add smoothing group * group smoothing in lib * correct changes * restore * add smoothing for mopitt * test data * change group name * update test data Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> * Feature/modis aod from h4 (#942) * change modis aod converter to take antive hdf4 files as input this requires installing the pyhdf python package * update test ref:wq * fix coding norms * fix cmakelist * pyhdf in ci * bug fix obsvar name * bugfix for variable name + testoutput Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: Maryam Abdi <maryamao@ucar.edu> * Add geometric height options for geovals (#1024) * A fix for SYNOP data containing BUFR subsets (#1021) * interim code, not yet debugged/working * handling subsets, is it even needed * placement of data into meta_data * much more refactoring, taking out repfacs that make zero sense in synop data * after more bug fixes to SYNOP converter * after more bug fixes to SYNOP converter * revert back reference file that uses bufr2ioda.x * bug fix for virtual temp and surface pres * fixing coding norms Co-authored-by: gthompsnJCSDA <gthompsn@ucar.edu> * Fix combine obsspace bug (#1027) Co-authored-by: Patrick Nichols <patjnichols@gmail.com> * Updated with ioda engines header file name change (Factory.h -> EngineUtils.h) (#1030) * Modified tests for MacOS SIP compatibility. (#1011) Co-authored-by: Patrick Nichols <patjnichols@gmail.com> * Changes in L2 trace gas retrievals converters for column operator generalization (#1025) * changes in iodaconv to make a better ufo L2 nadir operator * fix typo * change index * beging work on amf, to finish * move the amf calculation in ioda conv * update tests * fix coding norms * fix exec speed * remove amf variables and code clarity * update testinput * faster tropoomi converter execution * fix type in pressure * update test output * fix coding norms Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> * Added satbias converter for gmi_gpm. (#1042) * add ompslpnc for geos adas along with cleanup. (#1043) * add ompslpnc for geos adas along with cleanup. * fix coding norm. * tropomi better flags handling (#1037) * use better flags * coding norms * combine two bool arrays * Update src/compo/tropomi_no2_nc2ioda.py Co-authored-by: Fabio L R Diniz <45880035+fabiolrdiniz@users.noreply.github.com> * Update src/compo/tropomi_no2_nc2ioda.py Co-authored-by: Fabio L R Diniz <45880035+fabiolrdiniz@users.noreply.github.com> * Update src/compo/tropomi_no2_nc2ioda.py Co-authored-by: Fabio L R Diniz <45880035+fabiolrdiniz@users.noreply.github.com> Co-authored-by: Fabio L R Diniz <45880035+fabiolrdiniz@users.noreply.github.com> Co-authored-by: Patrick Nichols <patjnichols@gmail.com> * Add variable for GNSS bending angle data, save orginal GSI QC flags a… (#1038) * Add variable for GNSS bending angle data, save orginal GSI QC flags and other corrections, and remove VarMetaData. * Removed the change of 'GsiEffectiveQC'. * Trigger a test. * Fixed coding norms. * Fixed coding norms. Again Co-authored-by: Dan Holdaway <27729500+danholdaway@users.noreply.github.com> * Replaced "land_type_index" by "land_type_index_NPOESS" (#921) * Replaced "land_type_index" by "land_type_index_NPOESS" in "gsi_ncdiag.py". * Updated gsi_ncdiag.py. * Updated gsi_ncdiag.py. * Removed Forecast_adjusted and Forecast_unadjusted for GSI radiance outputs. * Moved some radiance variables into the new group VarMetaData. * Add variable for GNSS bending angle data, save orginal GSI QC flags and other corrections, and remove VarMetaData. * Only changed land_type_index to land_type_index_NPOESS. Co-authored-by: Patrick Nichols <patjnichols@gmail.com> * add capability to create MOPITT IODA files for DA window (#1039) * save * add test and change converter * Update src/compo/mopitt_co_nc2ioda.py Co-authored-by: Fabio L R Diniz <45880035+fabiolrdiniz@users.noreply.github.com> * Update src/compo/mopitt_co_nc2ioda.py Co-authored-by: Fabio L R Diniz <45880035+fabiolrdiniz@users.noreply.github.com> * Update src/compo/mopitt_co_nc2ioda.py Co-authored-by: Fabio L R Diniz <45880035+fabiolrdiniz@users.noreply.github.com> * change to time time range Co-authored-by: Fabio L R Diniz <45880035+fabiolrdiniz@users.noreply.github.com> Co-authored-by: Patrick Nichols <patjnichols@gmail.com> * Feature/query cxx (#1006) * First cut at calling the new nceplibs bufr query interface. * Updated to handle multi-dimensional data. Fixed DatetimeVariable. Got bufr mhs example to work. * Added support for CCITT IA5 string fields. Fixed many other fields that not not properly support queries. * Added support for forfield. Cleaned up some uneccesarrily complicated data structures. * Removed split map in favor of a list of splits to make code more readable. * Fixed bug with transforms. Updated some YAML files. * Updated the README file. * Fixed bug were CCITT IA5 fields would fail because of a silly mistake (throwing exception unneccessarliy). * Renamed for field to group_by field. * Removed duplicate directory for transforms. * incremental checkin, getting higher dimensional data working * Added DataObject cpp * Finished major code revisions and got things to compile for automatic dimensioning. * Some bug fixes to get simple radiance data working. * Enhanced the reading of th dimensions field. * Got filtering and splitting to work again. Might need to be a little more careful with slice dimensions, but works with existing examples. * Fixed a dimension naming issue with the way group_by location dimensions were being named. Fixed some compiler warnings. * Added arguments to bufr2ioda so you can limit the amount of messages that are parsed. * Changed dimensions attribute to list or single string. * Fixed transforms. * Fixed transfoms so they will not affect missing values. * Fixed bug that happened when we tried to export an empty category. * Fixed strange issue that happened while applying filters. Removed some redundant yaml files from testinput. * Updated a bunch of unit tests so that they would work with the latest code. * updated test output files to work with latest version of ioda * Added unit test and now time the runtime. * Fixed a couple more unit tests. * Added .dump files to gitlfs * removed uneeded file * BUFR2IODA conversion for AMSU-A and MHS Radiance data (#713) * Add bufr to ioda conversion for IASI, ERAS/DB AMSU-A and MHS for ncar-bufr2nc-fortarn utility * Add tests and associated test input/output files for ncar-bufr2nc-fortarn utility * Add NCEP BUFR tables for normal-feed and RARS/DB satellite radiance data * Add YAML files, tests, and test reference for normal-feed and db/ears AMSU-A * Fix coding norm * Rename NCEP BUFR tables for satellite radiance data * Add yaml files, tests, and test input/output files for MHS (normal-feed and RARS/DB) t * Update Split.cpp * Update DataObject.cpp * Update IodaEncoder.cpp * Update bufr_ncep_1bamua.yaml * Update bufr_ncep_1bmhs.yaml * Update bufr_ncep_esamua.yaml * Update bufr_ncep_esmhs.yaml * Fixed coding norms issue. * Fixed some little issues. Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Update README.md Updated documentation for dimensions. * Test and enhance two yaml files for adpsfc snow and sevcrs data (#736) * Update snowadpsfc * Update bufr_snow_adpsfc.yaml * Update testoutput * Update yaml files and testinput and testoutput data * Reduce input data size to 20% and update output * Reduce input data size to 250KB and update testoutput * Reduce the size of test files for amsua, mhs and iasi. Update the corresponding test output files (#764) * Removed uneeded files I missed during merge. * Initial checkin of query classes with a lot of placeholder code * Got data collection mostly working. * Starting to work end to end. Need to fix exportdims and revisit code that deals with group by fields. * Fixed issue where targets and masks were being copied unecessarily. * Feature/adpupa prepbufr (#794) * Added adpupa prepbufr yamls w/o group_by field with input adpupa prepbufr file * adpupa netCDF files w/o group_by field * Updates made after ioda-validate.x run. * Changed Quality Mark group from "ObsQualityMark" to "QualityMarker" * Got a bunch of unit tests working. * add aircft profiles to feature/query * Fixed non-working functional unit tests. * Added unit test for simple groupby example. * Made simple groupby test work. Fixed related bugs. * Added unit test for bufr files with missing fields. * Added unit test for sfcshp data * accidentaly messed up the bufr-mhs test. this fixes it * Integrated latest bug fixes from the feature/query branch. * Fixed bug were we weren't checking for empty sequence counts. * added adpupa tests * Fixed bugs that prevented adpupa from working. * Getting much closer to having adpupa groupby example working. * fixed final bug to get adpupa groupby case to work the same way. * Removed some uneeded code. * Reduce the size of the ADPUPA test input furrther * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. Fixed some copyright statements. * Changed the way data provider works (no longer a singleton). Minor cleanup. * Added print_queries tool to the ioda-converter project. * Improved situation arrising from missing string fields. * Improved detection of string fields. * Moved table logic out of the print_quiries function. Fixed bug were idxs were not being displayed on queries. * Replace nlocs with Location, reduce the size of ADPUPA.prepbufr and … (#884) * Replace nlocs with Location. Reduce the size of ADPUPA.prepbufr and update the test output. * remove checking of satellite id in QC (#878) * remove checking of satellite id in QC * added checking of bits 1 and 4 in QC flag Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> * Feature/optional python qc (#881) * Corrected bit checks due to reverse ordered bits * add check if there any data to write out * update ctest input and reference file * add new test output verification * corrected pycodestyle error * added optional georeality QC checks similar to those present in FORTRAN converter * pycodestyle corrections * removed percent confidence check * attempting fix of conflicts with existing code * corrected syntax errors * removed global variable Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: haydenlj <99764299+haydenlj@users.noreply.github.com> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Reverted ADPUPA bufr file to smaller one. * Fixed compiler warnings * Fixed code style (norm) issues. * Added doc strings to query code. * Feature/query cxx praveen (#893) * adpupa prepbufr yaml for updated group_by field and corresponding testoutput file * adpupa prepbufr yaml containing GSI mnemonics only and corresponding output file * adpupa gsi prepbufr file with updated name, groups, units, with bufr2ioda_cxx.x generated netCDF file * added TVO and CLTP to the adpupa prepbufr yaml, updated name and output netcdf files * Updated dimensions into yaml * Added group_by fields for ELV, TYP, and T29; and updated the nc files * Added white space * Change dimension name to following unified IODA naming convention (#767) * Reduce the size of test files for amsua, mhs and iasi. Update the corresponding test output files * Replace "nlocs" with "Location" and "nchans" with "Channel" (to following the unified IODA naming convention). * Fix a typo and update test output accordingly * Added white space * Made updates to make everything consistent with query_cxx branch Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Fixed bug when there were more than one subset in the file. Added ability to print monotone output (so you can pipe to a text file). * Updated print queries so it no longer uses colors but prints the dimensioning subpaths seperatly. * Chenged some maps to unordered_maps as they are better performance wise. * Feature/query cxx as (#914) * Updated yaml, yamlinput, and yaml output for SatWind AVHRR NC005080 within the new feature/query_cxx branch. * The bufr_ncep_sawtind_avhrr yaml has been updated to include and produce input and output with the same name scheme. Files have been renamed accordingly. * Modified input and output fields of the yaml file. * modified: CMakeLists.txt renamed and modified: bufr_ncep_satwind_avhrr.yaml -> testinput/bufr_ncep_satwind_avhrr.yaml The input fields and linked name fields for "time" has been updated as suggested. The CMakeList has been updated to match with the new naming scheme. The "dimensions" field has been updated as suggested. And the yaml has now been moved from ./test to ./test/testinput. * Removed indices for most variables, while adding in 2 more indices for GNAP and PCCF to get wind direction and wind speed generating application and percent confidence. * Updated variables names to match the JEDI convention spreadsheet. * Removed a default/backup configuration line. * Modify unit attribute in BUFR2IODA converter and bug fix for satellite winds (#940) * Make units attribute an optional variable * Bug fix for AVHRR satellite wind test: (1) unit in yaml (2) output directory in yaml (3) test output file in CMakelist.txt * Change the test input and output file names according to the following logic: (1) for input, use the global dump file naming convention (e.g. gdas.t00z.satwnd.tm00.bufr_d) (2) for output change the input file name extension bufr_d to nc (gdas.t00z.satwnd.tm00.nc) * delete bufr_ncep_satwind_avhrr.nc * Make sure the time info is consistent between the file name and the data in the file. * adjust the order of files listed in the CMakeList.txt * Change the way to test the empty string. * Fixed issue where non-existing query with index would cause a crash. Also fixed bug where the suggested dimesioning paths would be messed up sometimes. * Fixed some norm test failures. * Fixed all the unit tests * Made change so that there is always one group_by field whcih applies to all the variables. * Removed uneeded test * Improved some of the code to do with finding the data object to group against. * Fixed bug in print queries were fixed repeats were being missed. * update two yaml files via query_cxx (#906) * update snow adpsfc yaml and the result * Update sevcsr yaml file and the result * Update bufr_ncep_read_2_dim_blocks.yaml * Fix dateTim with DateTime, nchans with Channel * Remove ncep for yaml and modify the CMakeLists.txt * Update bufr_ncep_snow_adpsfc.yaml * Update bufr_read_2_dim_blocks.yaml to follow JEDI convention and update the associated test output (bufr_read_2_dim_blocks.nc). * Modified yaml, add sacle and update the result * Changes cloudAmount unit and update the result * Change cloudAmount to cloudCoverTotal * Set BUFR variable: CLDMNT to cloudAmount since it is already in JEDI convention with optional dimenion as [location, channel] Co-authored-by: Emily Liu <emily.liu@noaa.gov> * Feature/query cxx mhs (#960) * Update 1bmhs YAML and test output * Update DB mhs YAML file and test output * Modify longnames and update test output files * Feature/query cxx amsua (#958) * Update YAMLS and test output files for 1bamsua and esamsua * Update YAML files for AMSU-A * Remove unwanted test file for amsua-a * fix spacing. * add spacings between lines * Change height to heightOfStation and update test output * Fix longnames * Feature/query cxx iasi (#966) * Update IASI yaml and related testoutput * Update IASI YAML * Update IASI test file * Fix a few errors in IASI YAML and update the test output * Feature/query cxx nick e adpsfc (#904) * adpsfc prepbufr and bufr * recommended changes * recommended updates 2 (* and deg) * CMakelists.txt update * update CMakeLists.txt * adpsfc prepbufr recommended changes * remove nc, add full adpsfcyaml * adpsfc updates, plus CMakelists * remove CLTP and VSSO * decrease number of variables * update output * heightOfStation -> stationElevation * get rid of oceanographic * update heightofbaseofcloud description * update with correct files * updates to HOCB CLAM VSSO * update thinned, has all subsets * Fixed the unit tests * reportid -> SID, CLAM[1] is CLAM, pW@Obs, percent ->1 * update output * new output from updated bufr2ioda.x * bufr: HSMSL+SELV=stationElevation, PB: percent->1 * get rid of units:"" and unitless. * add dimensions. cloudAmountObserved. cloudCover/100 * moistureQM -> dewpointTemperatureQM * cloudAmount no units * updated bufr with new dim names * update prep. few names. add dimens * most updates for prep * No more 'Seen'. Other updates * delete metarsequence, Of not of * Move VSSO to be with other variables related to clouds * Rearrange the order of entries in the YAML * fix spacing in the YAML * Reorder entries in YAML * fix spaceing * Rearrange entries * Modify variable names and bug fix for ADPSFC BUFR and PREPBUFR YAMLS. Test output files also updated. * Update CMakelist.txt and add comments * Remove obselete files * Modify ADPSFC test output back to original * Modify variable names for max/min Temperatures * Add missing variables and rearranged entries. * Update test output for ADPSFC * Change heightOfStation to heightOfObservation * Update ctest output file for prepbufr adpsfc * get rid of v1 adpsfc. final recommended updates * revert, keep gdas.t06z.adpsfc.tm00.bufr_d for adpsfc_snow * re-add adpsfc_snow input bufr to CmakeLists.txt * Add the correct input test file for adpsfc snow data * update prepbufr file names, CMake, yaml obsdatain and out * typo * degrees_north and east -> degree_north and east * delete 'adsfc' file, update with degree Co-authored-by: Emily Liu <emily.liu@noaa.gov> Co-authored-by: Ron McLaren <ronlaren@aol.com> * Feature/query cxx atovs (#963) * Update test input files so data from all related satellite platform are included. Rearrange the entries in the YAML files. * update tes output files so that the naming convention of the filename is consistent. * Update test output for iodaconv fortran * Updated DatetimeVariable.cpp code for optional value of Minute (#949) * Updated DatetimeVariable.cpp code for optional value of Minute * added a yaml file to test the DatetimeVariable.cpp code for optional MINU variable Co-authored-by: Emily Liu <emily.liu@noaa.gov> * Now uses OOPS logging. * Fixed some comments * cast time to seconds instead of dividing by 1000 * fixed norm test failure * made exit into exception * using back inserter to append to sequence path * fixed error message. * fixed error message. * added prepbufr to git lfs * removed line continuation characters. * replaced 10e10 with missing value * accidentally deleted line of code. added it back * changed print time function back tot he original. * using gsl::narrow instead of simpoe static cast * Enhanced vector product function * Pulled out missing value into a Constants.h file to avoid duplication. * fixed some comments * fixed file permissions * Added try catch statements to main functions. Fixed warning. * pre reserve some memory * removed unused test from CMakeLists.txt file. * Added include of unordered map as some compilers were not finding it. * Reverted unintended changes to files in ncar-bufr2nc-fortran and lib-python * Fixed bug that invovled parsing the bufr table data incorrectly. Basically we were not exiting trivial sequences properly when parsing the table. * Fixed bug in print queries as it was not trimming subset trims properly and was getting confused. * Fixed some warnings and errors that happen with the CLANG compiler. * Enable the C++14 standard for C++ compiles. (#1036) * Fixed some code review issues that Cory Martin broght up. * updated docs for DataObject::print function * commented in line that was accidentally commented out * Revert "commented in line that was accidentally commented out" This reverts commit c78b93eae745b64b45567cec0ec2388ec129d0fb. * Added back missing tests from the develop branch. * Moved test_iodaconv_generic_gnssro_bufr back to where it was before (positionaly). * Fixed issue caused by code review comment fix that caused bufr2ioda.x to obfuscate excepotion descriptions. * Fixed exception handler in print_queries.x. * Fixed Mac SIP issues. * Commented out Emily Liu's bufr2nc_fortran.x tests.. * trigger ci * changes needed for new containers (#1056) * changes needed for new containers * debug * debug * debug * First stab at fixing CI using new containers * Fix jedi-cmake * Try make instead of ctest * Try ctest after make * Try ctest -j4 for intel * Try ctest -j8 for intel * Remove comments from buildspecs * More cleanup in CI/buildspec_gnu.yml * Fix CDASH links Co-authored-by: Dom Heinzeller <dom.heinzeller@icloud.com> Co-authored-by: Emily Liu <emily.liu@noaa.gov> Co-authored-by: emilyhcliu <36091766+emilyhcliu@users.noreply.github.com> Co-authored-by: YoulongXia-NOAA <51668688+YoulongXia-NOAA@users.noreply.github.com> Co-authored-by: PraveenKumar-NOAA <63739712+PraveenKumar-NOAA@users.noreply.github.com> Co-authored-by: Nicholas.Esposito <Nicholas.Esposito@noaa.gov> Co-authored-by: haydenlj <99764299+haydenlj@users.noreply.github.com> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: AshleyStanfield-NOAA <92459593+AshleyStanfield-NOAA@users.noreply.github.com> Co-authored-by: NicholasEsposito-NOAA <62616739+nicholasesposito@users.noreply.github.com> Co-authored-by: Stephen Herbener <stephen.herbener@gmail.com> Co-authored-by: Cory Martin <cory.r.martin@noaa.gov> Co-authored-by: Patrick Nichols <patjnichols@gmail.com> Co-authored-by: Maryam Abdi-Oskouei <maryamao@ucar.edu> Co-authored-by: Dom Heinzeller <dom.heinzeller@icloud.com> * change from two composition related ctest to nonzero tolerances (#1064) * bugfix (#1065) Co-authored-by: Dom Heinzeller <dom.heinzeller@icloud.com> * fix typo index (#1057) Co-authored-by: Maryam Abdi <maryamao@ucar.edu> Co-authored-by: Dom Heinzeller <dom.heinzeller@icloud.com> * Feature/query dim naming bug (#1040) * First cut at calling the new nceplibs bufr query interface. * Updated to handle multi-dimensional data. Fixed DatetimeVariable. Got bufr mhs example to work. * Added support for CCITT IA5 string fields. Fixed many other fields that not not properly support queries. * Added support for forfield. Cleaned up some uneccesarrily complicated data structures. * Removed split map in favor of a list of splits to make code more readable. * Fixed bug with transforms. Updated some YAML files. * Updated the README file. * Fixed bug were CCITT IA5 fields would fail because of a silly mistake (throwing exception unneccessarliy). * Renamed for field to group_by field. * Removed duplicate directory for transforms. * incremental checkin, getting higher dimensional data working * Added DataObject cpp * Finished major code revisions and got things to compile for automatic dimensioning. * Some bug fixes to get simple radiance data working. * Enhanced the reading of th dimensions field. * Got filtering and splitting to work again. Might need to be a little more careful with slice dimensions, but works with existing examples. * Fixed a dimension naming issue with the way group_by location dimensions were being named. Fixed some compiler warnings. * Added arguments to bufr2ioda so you can limit the amount of messages that are parsed. * Changed dimensions attribute to list or single string. * Fixed transforms. * Fixed transfoms so they will not affect missing values. * Fixed bug that happened when we tried to export an empty category. * Fixed strange issue that happened while applying filters. Removed some redundant yaml files from testinput. * Updated a bunch of unit tests so that they would work with the latest code. * updated test output files to work with latest version of ioda * Added unit test and now time the runtime. * Fixed a couple more unit tests. * Added .dump files to gitlfs * removed uneeded file * BUFR2IODA conversion for AMSU-A and MHS Radiance data (#713) * Add bufr to ioda conversion for IASI, ERAS/DB AMSU-A and MHS for ncar-bufr2nc-fortarn utility * Add tests and associated test input/output files for ncar-bufr2nc-fortarn utility * Add NCEP BUFR tables for normal-feed and RARS/DB satellite radiance data * Add YAML files, tests, and test reference for normal-feed and db/ears AMSU-A * Fix coding norm * Rename NCEP BUFR tables for satellite radiance data * Add yaml files, tests, and test input/output files for MHS (normal-feed and RARS/DB) t * Update Split.cpp * Update DataObject.cpp * Update IodaEncoder.cpp * Update bufr_ncep_1bamua.yaml * Update bufr_ncep_1bmhs.yaml * Update bufr_ncep_esamua.yaml * Update bufr_ncep_esmhs.yaml * Fixed coding norms issue. * Fixed some little issues. Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Update README.md Updated documentation for dimensions. * Test and enhance two yaml files for adpsfc snow and sevcrs data (#736) * Update snowadpsfc * Update bufr_snow_adpsfc.yaml * Update testoutput * Update yaml files and testinput and testoutput data * Reduce input data size to 20% and update output * Reduce input data size to 250KB and update testoutput * Reduce the size of test files for amsua, mhs and iasi. Update the corresponding test output files (#764) * Removed uneeded files I missed during merge. * Initial checkin of query classes with a lot of placeholder code * Got data collection mostly working. * Starting to work end to end. Need to fix exportdims and revisit code that deals with group by fields. * Fixed issue where targets and masks were being copied unecessarily. * Feature/adpupa prepbufr (#794) * Added adpupa prepbufr yamls w/o group_by field with input adpupa prepbufr file * adpupa netCDF files w/o group_by field * Updates made after ioda-validate.x run. * Changed Quality Mark group from "ObsQualityMark" to "QualityMarker" * Got a bunch of unit tests working. * add aircft profiles to feature/query * Fixed non-working functional unit tests. * Added unit test for simple groupby example. * Made simple groupby test work. Fixed related bugs. * Added unit test for bufr files with missing fields. * Added unit test for sfcshp data * accidentaly messed up the bufr-mhs test. this fixes it * Integrated latest bug fixes from the feature/query branch. * Fixed bug were we weren't checking for empty sequence counts. * added adpupa tests * Fixed bugs that prevented adpupa from working. * Getting much closer to having adpupa groupby example working. * fixed final bug to get adpupa groupby case to work the same way. * Removed some uneeded code. * Reduce the size of the ADPUPA test input furrther * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. Fixed some copyright statements. * Changed the way data provider works (no longer a singleton). Minor cleanup. * Added print_queries tool to the ioda-converter project. * Improved situation arrising from missing string fields. * Improved detection of string fields. * Moved table logic out of the print_quiries function. Fixed bug were idxs were not being displayed on queries. * Replace nlocs with Location, reduce the size of ADPUPA.prepbufr and … (#884) * Replace nlocs with Location. Reduce the size of ADPUPA.prepbufr and update the test output. * remove checking of satellite id in QC (#878) * remove checking of satellite id in QC * added checking of bits 1 and 4 in QC flag Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> * Feature/optional python qc (#881) * Corrected bit checks due to reverse ordered bits * add check if there any data to write out * update ctest input and reference file * add new test output verification * corrected pycodestyle error * added optional georeality QC checks similar to those present in FORTRAN converter * pycodestyle corrections * removed percent confidence check * attempting fix of conflicts with existing code * corrected syntax errors * removed global variable Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: haydenlj <99764299+haydenlj@users.noreply.github.com> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Reverted ADPUPA bufr file to smaller one. * Fixed compiler warnings * Fixed code style (norm) issues. * Added doc strings to query code. * Feature/query cxx praveen (#893) * adpupa prepbufr yaml for updated group_by field and corresponding testoutput file * adpupa prepbufr yaml containing GSI mnemonics only and corresponding output file * adpupa gsi prepbufr file with updated name, groups, units, with bufr2ioda_cxx.x generated netCDF file * added TVO and CLTP to the adpupa prepbufr yaml, updated name and output netcdf files * Updated dimensions into yaml * Added group_by fields for ELV, TYP, and T29; and updated the nc files * Added white space * Change dimension name to following unified IODA naming convention (#767) * Reduce the size of test files for amsua, mhs and iasi. Update the corresponding test output files * Replace "nlocs" with "Location" and "nchans" with "Channel" (to following the unified IODA naming convention). * Fix a typo and update test output accordingly * Added white space * Made updates to make everything consistent with query_cxx branch Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Fixed bug when there were more than one subset in the file. Added ability to print monotone output (so you can pipe to a text file). * Updated print queries so it no longer uses colors but prints the dimensioning subpaths seperatly. * Chenged some maps to unordered_maps as they are better performance wise. * Feature/query cxx as (#914) * Updated yaml, yamlinput, and yaml output for SatWind AVHRR NC005080 within the new feature/query_cxx branch. * The bufr_ncep_sawtind_avhrr yaml has been updated to include and produce input and output with the same name scheme. Files have been renamed accordingly. * Modified input and output fields of the yaml file. * modified: CMakeLists.txt renamed and modified: bufr_ncep_satwind_avhrr.yaml -> testinput/bufr_ncep_satwind_avhrr.yaml The input fields and linked name fields for "time" has been updated as suggested. The CMakeList has been updated to match with the new naming scheme. The "dimensions" field has been updated as suggested. And the yaml has now been moved from ./test to ./test/testinput. * Removed indices for most variables, while adding in 2 more indices for GNAP and PCCF to get wind direction and wind speed generating application and percent confidence. * Updated variables names to match the JEDI convention spreadsheet. * Removed a default/backup configuration line. * Modify unit attribute in BUFR2IODA converter and bug fix for satellite winds (#940) * Make units attribute an optional variable * Bug fix for AVHRR satellite wind test: (1) unit in yaml (2) output directory in yaml (3) test output file in CMakelist.txt * Change the test input and output file names according to the following logic: (1) for input, use the global dump file naming convention (e.g. gdas.t00z.satwnd.tm00.bufr_d) (2) for output change the input file name extension bufr_d to nc (gdas.t00z.satwnd.tm00.nc) * delete bufr_ncep_satwind_avhrr.nc * Make sure the time info is consistent between the file name and the data in the file. * adjust the order of files listed in the CMakeList.txt * Change the way to test the empty string. * Fixed issue where non-existing query with index would cause a crash. Also fixed bug where the suggested dimesioning paths would be messed up sometimes. * Fixed some norm test failures. * Fixed all the unit tests * Made change so that there is always one group_by field whcih applies to all the variables. * Removed uneeded test * Improved some of the code to do with finding the data object to group against. * Fixed bug in print queries were fixed repeats were being missed. * update two yaml files via query_cxx (#906) * update snow adpsfc yaml and the result * Update sevcsr yaml file and the result * Update bufr_ncep_read_2_dim_blocks.yaml * Fix dateTim with DateTime, nchans with Channel * Remove ncep for yaml and modify the CMakeLists.txt * Update bufr_ncep_snow_adpsfc.yaml * Update bufr_read_2_dim_blocks.yaml to follow JEDI convention and update the associated test output (bufr_read_2_dim_blocks.nc). * Modified yaml, add sacle and update the result * Changes cloudAmount unit and update the result * Change cloudAmount to cloudCoverTotal * Set BUFR variable: CLDMNT to cloudAmount since it is already in JEDI convention with optional dimenion as [location, channel] Co-authored-by: Emily Liu <emily.liu@noaa.gov> * Feature/query cxx mhs (#960) * Update 1bmhs YAML and test output * Update DB mhs YAML file and test output * Modify longnames and update test output files * Feature/query cxx amsua (#958) * Update YAMLS and test output files for 1bamsua and esamsua * Update YAML files for AMSU-A * Remove unwanted test file for amsua-a * fix spacing. * add spacings between lines * Change height to heightOfStation and update test output * Fix longnames * Feature/query cxx iasi (#966) * Update IASI yaml and related testoutput * Update IASI YAML * Update IASI test file * Fix a few errors in IASI YAML and update the test output * Feature/query cxx nick e adpsfc (#904) * adpsfc prepbufr and bufr * recommended changes * recommended updates 2 (* and deg) * CMakelists.txt update * update CMakeLists.txt * adpsfc prepbufr recommended changes * remove nc, add full adpsfcyaml * adpsfc updates, plus CMakelists * remove CLTP and VSSO * decrease number of variables * update output * heightOfStation -> stationElevation * get rid of oceanographic * update heightofbaseofcloud description * update with correct files * updates to HOCB CLAM VSSO * update thinned, has all subsets * Fixed the unit tests * reportid -> SID, CLAM[1] is CLAM, pW@Obs, percent ->1 * update output * new output from updated bufr2ioda.x * bufr: HSMSL+SELV=stationElevation, PB: percent->1 * get rid of units:"" and unitless. * add dimensions. cloudAmountObserved. cloudCover/100 * moistureQM -> dewpointTemperatureQM * cloudAmount no units * updated bufr with new dim names * update prep. few names. add dimens * most updates for prep * No more 'Seen'. Other updates * delete metarsequence, Of not of * Move VSSO to be with other variables related to clouds * Rearrange the order of entries in the YAML * fix spacing in the YAML * Reorder entries in YAML * fix spaceing * Rearrange entries * Modify variable names and bug fix for ADPSFC BUFR and PREPBUFR YAMLS. Test output files also updated. * Update CMakelist.txt and add comments * Remove obselete files * Modify ADPSFC test output back to original * Modify variable names for max/min Temperatures * Add missing variables and rearranged entries. * Update test output for ADPSFC * Change heightOfStation to heightOfObservation * Update ctest output file for prepbufr adpsfc * get rid of v1 adpsfc. final recommended updates * revert, keep gdas.t06z.adpsfc.tm00.bufr_d for adpsfc_snow * re-add adpsfc_snow input bufr to CmakeLists.txt * Add the correct input test file for adpsfc snow data * update prepbufr file names, CMake, yaml obsdatain and out * typo * degrees_north and east -> degree_north and east * delete 'adsfc' file, update with degree Co-authored-by: Emily Liu <emily.liu@noaa.gov> Co-authored-by: Ron McLaren <ronlaren@aol.com> * Feature/query cxx atovs (#963) * Update test input files so data from all related satellite platform are included. Rearrange the entries in the YAML files. * update tes output files so that the naming convention of the filename is consistent. * Update test output for iodaconv fortran * Updated DatetimeVariable.cpp code for optional value of Minute (#949) * Updated DatetimeVariable.cpp code for optional value of Minute * added a yaml file to test the DatetimeVariable.cpp code for optional MINU variable Co-authored-by: Emily Liu <emily.liu@noaa.gov> * Now uses OOPS logging. * Fixed some comments * cast time to seconds instead of dividing by 1000 * fixed norm test failure * made exit into exception * using back inserter to append to sequence path * fixed error message. * fixed error message. * added prepbufr to git lfs * removed line continuation characters. * replaced 10e10 with missing value * accidentally deleted line of code. added it back * changed print time function back tot he original. * using gsl::narrow instead of simpoe static cast * Enhanced vector product function * Pulled out missing value into a Constants.h file to avoid duplication. * fixed some comments * fixed file permissions * Added try catch statements to main functions. Fixed warning. * pre reserve some memory * removed unused test from CMakeLists.txt file. * Added include of unordered map as some compilers were not finding it. * Reverted unintended changes to files in ncar-bufr2nc-fortran and lib-python * Fixed bug that invovled parsing the bufr table data incorrectly. Basically we were not exiting trivial sequences properly when parsing the table. * Fixed bug in print queries as it was not trimming subset trims properly and was getting confused. * Fixed some warnings and errors that happen with the CLANG compiler. * Enable the C++14 standard for C++ compiles. (#1036) * Fixed some code review issues that Cory Martin broght up. * updated docs for DataObject::print function * commented in line that was accidentally commented out * Revert "commented in line that was accidentally commented out" This reverts commit c78b93eae745b64b45567cec0ec2388ec129d0fb. * Added back missing tests from the develop branch. * Moved test_iodaconv_generic_gnssro_bufr back to where it was before (positionaly). * Forcing named dimensions to be generic paths (not based on subset). Basically they must start with *. * Fixed issue caused by code review comment fix that caused bufr2ioda.x to obfuscate excepotion descriptions. * Fixed exception handler in print_queries.x. Co-authored-by: Emily Liu <emily.liu@noaa.gov> Co-authored-by: emilyhcliu <36091766+emilyhcliu@users.noreply.github.com> Co-authored-by: YoulongXia-NOAA <51668688+YoulongXia-NOAA@users.noreply.github.com> Co-authored-by: PraveenKumar-NOAA <63739712+PraveenKumar-NOAA@users.noreply.github.com> Co-authored-by: Nicholas.Esposito <Nicholas.Esposito@noaa.gov> Co-authored-by: haydenlj <99764299+haydenlj@users.noreply.github.com> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: AshleyStanfield-NOAA <92459593+AshleyStanfield-NOAA@users.noreply.github.com> Co-authored-by: NicholasEsposito-NOAA <62616739+nicholasesposito@users.noreply.github.com> Co-authored-by: Stephen Herbener <stephen.herbener@gmail.com> Co-authored-by: Cory Martin <cory.r.martin@noaa.gov> Co-authored-by: Patrick Nichols <patjnichols@gmail.com> * ioda converter for synthetic SWOT observations (#1058) * Adding SWOT ADT converter * adding swot l2 ssh 2 adt ioda converter and test data * adding the data to CMakelists * Reducing test file and updating adt estimate * fix code norms Co-authored-by: Kriti <> Co-authored-by: Travis Sluka <travissluka@gmail.com> Co-authored-by: Benjamin Ruston <93737224+BenjaminRuston@users.noreply.github.com> * Fix reference to oz_sensors in proc_gsi_ncdiag (#1053) * Fix oz_sensor issue * geoval fix too * empty commit to trigger CI as ufo-data was out of sync Co-authored-by: Benjamin Ruston <93737224+BenjaminRuston@users.noreply.github.com> Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: Fabio L R Diniz <45880035+fabiolrdiniz@users.noreply.github.com> * Add ability to convert Text radiosonde data (TEMP format) to IODA (#1060) * initial checkin of code for converter for text (TAC) radiosonde to ioda * fixed many pieces including proper output of IODA file * adding in new share subdir and the station file for radiosondes, json format * add in small raob_stations file for rapid ctest * addition of testoutput ref file * add ctest for sonde tac2ioda and also add share mkdir and linking * fixing coding norms * add the sonde_tac2ioda.py file to CMake * need to add another utils file * fix path for since it needs to be relative to test * another update to relative path for share * forgot the testinput small file * fix testoutput filename for linking * update ref file with changed attribute of source file Co-authored-by: gthompsnJCSDA <gthompsn@ucar.edu> Co-authored-by: Benjamin Ruston <93737224+BenjaminRuston@users.noreply.github.com> * Updated viirs_aod2ioda.py to include nchans for channel number (#1046) * Updated viirs_aod2ioda.py to include nchans for channel number * Fixed coding norms Co-authored-by: Patrick Nichols <patjnichols@gmail.com> Co-authored-by: Benjamin Ruston <93737224+BenjaminRuston@users.noreply.github.com> * Feature/query cxx types (#1044) * First cut at calling the new nceplibs bufr query interface. * Updated to handle multi-dimensional data. Fixed DatetimeVariable. Got bufr mhs example to work. * Added support for CCITT IA5 string fields. Fixed many other fields that not not properly support queries. * Added support for forfield. Cleaned up some uneccesarrily complicated data structures. * Removed split map in favor of a list of splits to make code more readable. * Fixed bug with transforms. Updated some YAML files. * Updated the README file. * Fixed bug were CCITT IA5 fields would fail because of a silly mistake (throwing exception unneccessarliy). * Renamed for field to group_by field. * Removed duplicate directory for transforms. * incremental checkin, getting higher dimensional data working * Added DataObject cpp * Finished major code revisions and got things to compile for automatic dimensioning. * Some bug fixes to get simple radiance data working. * Enhanced the reading of th dimensions field. * Got filtering and splitting to work again. Might need to be a little more careful with slice dimensions, but works with existing examples. * Fixed a dimension naming issue with the way group_by location dimensions were being named. Fixed some compiler warnings. * Added arguments to bufr2ioda so you can limit the amount of messages that are parsed. * Changed dimensions attribute to list or single string. * Fixed transforms. * Fixed transfoms so they will not affect missing values. * Fixed bug that happened when we tried to export an empty category. * Fixed strange issue that happened while applying filters. Removed some redundant yaml files from testinput. * Updated a bunch of unit tests so that they would work with the latest code. * updated test output files to work with latest version of ioda * Added unit test and now time the runtime. * Fixed a couple more unit tests. * Added .dump files to gitlfs * removed uneeded file * BUFR2IODA conversion for AMSU-A and MHS Radiance data (#713) * Add bufr to ioda conversion for IASI, ERAS/DB AMSU-A and MHS for ncar-bufr2nc-fortarn utility * Add tests and associated test input/output files for ncar-bufr2nc-fortarn utility * Add NCEP BUFR tables for normal-feed and RARS/DB satellite radiance data * Add YAML files, tests, and test reference for normal-feed and db/ears AMSU-A * Fix coding norm * Rename NCEP BUFR tables for satellite radiance data * Add yaml files, tests, and test input/output files for MHS (normal-feed and RARS/DB) t * Update Split.cpp * Update DataObject.cpp * Update IodaEncoder.cpp * Update bufr_ncep_1bamua.yaml * Update bufr_ncep_1bmhs.yaml * Update bufr_ncep_esamua.yaml * Update bufr_ncep_esmhs.yaml * Fixed coding norms issue. * Fixed some little issues. Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Update README.md Updated documentation for dimensions. * Test and enhance two yaml files for adpsfc snow and sevcrs data (#736) * Update snowadpsfc * Update bufr_snow_adpsfc.yaml * Update testoutput * Update yaml files and testinput and testoutput data * Reduce input data size to 20% and update output * Reduce input data size to 250KB and update testoutput * Reduce the size of test files for amsua, mhs and iasi. Update the corresponding test output files (#764) * Removed uneeded files I missed during merge. * Initial checkin of query classes with a lot of placeholder code * Got data collection mostly working. * Starting to work end to end. Need to fix exportdims and revisit code that deals with group by fields. * Fixed issue where targets and masks were being copied unecessarily. * Feature/adpupa prepbufr (#794) * Added adpupa prepbufr yamls w/o group_by field with input adpupa prepbufr file * adpupa netCDF files w/o group_by field * Updates made after ioda-validate.x run. * Changed Quality Mark group from "ObsQualityMark" to "QualityMarker" * Got a bunch of unit tests working. * add aircft profiles to feature/query * Fixed non-working functional unit tests. * Added unit test for simple groupby example. * Made simple groupby test work. Fixed related bugs. * Added unit test for bufr files with missing fields. * Added unit test for sfcshp data * accidentaly messed up the bufr-mhs test. this fixes it * Integrated latest bug fixes from the feature/query branch. * Fixed bug were we weren't checking for empty sequence counts. * added adpupa tests * Fixed bugs that prevented adpupa from working. * Getting much closer to having adpupa groupby example working. * fixed final bug to get adpupa groupby case to work the same way. * Removed some uneeded code. * Reduce the size of the ADPUPA test input furrther * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. Fixed some copyright statements. * Changed the way data provider works (no longer a singleton). Minor cleanup. * Added print_queries tool to the ioda-converter project. * Improved situation arrising from missing string fields. * Improved detection of string fields. * Moved table logic out of the print_quiries function. Fixed bug were idxs were not being displayed on queries. * Replace nlocs with Location, reduce the size of ADPUPA.prepbufr and … (#884) * Replace nlocs with Location. Reduce the size of ADPUPA.prepbufr and update the test output. * remove checking of satellite id in QC (#878) * remove checking of satellite id in QC * added checking of bits 1 and 4 in QC flag Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> * Feature/optional python qc (#881) * Corrected bit checks due to reverse ordered bits * add check if there any data to write out * update ctest input and reference file * add new test output verification * corrected pycodestyle error * added optional georeality QC checks similar to those present in FORTRAN converter * pycodestyle corrections * removed percent confidence check * attempting fix of conflicts with existing code * corrected syntax errors * removed global variable Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: haydenlj <99764299+haydenlj@users.noreply.github.com> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Reverted ADPUPA bufr file to smaller one. * Fixed compiler warnings * Fixed code style (norm) issues. * Added doc strings to query code. * Feature/query cxx praveen (#893) * adpupa prepbufr yaml for updated group_by field and corresponding testoutput file * adpupa prepbufr yaml containing GSI mnemonics only and corresponding output file * adpupa gsi prepbufr file with updated name, groups, units, with bufr2ioda_cxx.x generated netCDF file * added TVO and CLTP to the adpupa prepbufr yaml, updated name and output netcdf files * Updated dimensions into yaml * Added group_by fields for ELV, TYP, and T29; and updated the nc files * Added white space * Change dimension name to following unified IODA naming convention (#767) * Reduce the size of test files for amsua, mhs and iasi. Update the corresponding test output files * Replace "nlocs" with "Location" and "nchans" with "Channel" (to following the unified IODA naming convention). * Fix a typo and update test output accordingly * Added white space * Made updates to make everything consistent with query_cxx branch Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Fixed bug when there were more than one subset in the file. Added ability to print monotone output (so you can pipe to a text file). * Updated print queries so it no longer uses colors but prints the dimensioning subpaths seperatly. * Chenged some maps to unordered_maps as they are better performance wise. * Feature/query cxx as (#914) * Updated yaml, yamlinput, and yaml output for SatWind AVHRR NC005080 within the new feature/query_cxx branch. * The bufr_ncep_sawtind_avhrr yaml has been updated to include and produce input and output with the same name scheme. Files have been renamed accordingly. * Modified input and output fields of the yaml file. * modified: CMakeLists.txt renamed and modified: bufr_ncep_satwind_avhrr.yaml -> testinput/bufr_ncep_satwind_avhrr.yaml The input fields and linked name fields for "time" has been updated as suggested. The CMakeList has been updated to match with the new naming scheme. The "dimensions" field has been updated as suggested. And the yaml has now been moved from ./test to ./test/testinput. * Removed indices for most variables, while adding in 2 more indices for GNAP and PCCF to get wind direction and wind speed generating application and percent confidence. * Updated variables names to match the JEDI convention spreadsheet. * Removed a default/backup configuration line. * Modify unit attribute in BUFR2IODA converter and bug fix for satellite winds (#940) * Make units attribute an optional variable * Bug fix for AVHRR satellite wind test: (1) unit in yaml (2) output directory in yaml (3) test output file in CMakelist.txt * Change the test input and output file names according to the following logic: (1) for input, use the global dump file naming convention (e.g. gdas.t00z.satwnd.tm00.bufr_d) (2) for output change the input file name extension bufr_d to nc (gdas.t00z.satwnd.tm00.nc) * delete bufr_ncep_satwind_avhrr.nc * Make sure the time info is consistent between the file name and the data in the file. * adjust the order of files listed in the CMakeList.txt * Change the way to test the empty string. * Fixed issue where non-existing query with index would cause a crash. Also fixed bug where the suggested dimesioning paths would be messed up sometimes. * Fixed some norm test failures. * Fixed all the unit tests * Made change so that there is always one group_by field whcih applies to all the variables. * Removed uneeded test * Improved some of the code to do with finding the data object to group against. * Fixed bug in print queries were fixed repeats were being missed. * update two yaml files via query_cxx (#906) * update snow adpsfc yaml and the result * Update sevcsr yaml file and the result * Update bufr_ncep_read_2_dim_blocks.yaml * Fix dateTim with DateTime, nchans with Channel * Remove ncep for yaml and modify the CMakeLists.txt * Update bufr_ncep_snow_adpsfc.yaml * Update bufr_read_2_dim_blocks.yaml to follow JEDI convention and update the associated test output (bufr_read_2_dim_blocks.nc). * Modified yaml, add sacle and update the result * Changes cloudAmount unit and update the result * Change cloudAmount to cloudCoverTotal * Set BUFR variable: CLDMNT to cloudAmount since it is already in JEDI convention with optional dimenion as [location, channel] Co-authored-by: Emily Liu <emily.liu@noaa.gov> * Feature/query cxx mhs (#960) * Update 1bmhs YAML and test output * Update DB mhs YAML file and test output * Modify longnames and update test output files * Feature/query cxx amsua (#958) * Update YAMLS and test output files for 1bamsua and esamsua * Update YAML files for AMSU-A * Remove unwanted test file for amsua-a * fix spacing. * add spacings between lines * Change height to heightOfStation and update test output * Fix longnames * Feature/query cxx iasi (#966) * Update IASI yaml and related testoutput * Update IASI YAML * Update IASI test file * Fix a few errors in IASI YAML and update the test output * Feature/query cxx nick e adpsfc (#904) * adpsfc prepbufr and bufr * recommended changes * recommended updates 2 (* and deg) * CMakelists.txt update * update CMakeLists.txt * adpsfc prepbufr recommended changes * remove nc, add full adpsfcyaml * adpsfc updates, plus CMakelists * remove CLTP and VSSO * decrease number of variables * update output * heightOfStation -> stationElevation * get rid of oceanographic * update heightofbaseofcloud description * update with correct files * updates to HOCB CLAM VSSO * update thinned, has all subsets * Fixed the unit tests * reportid -> SID, CLAM[1] is CLAM, pW@Obs, percent ->1 * update output * new output from updated bufr2ioda.x * bufr: HSMSL+SELV=stationElevation, PB: percent->1 * get rid of units:"" and unitless. * add dimensions. cloudAmountObserved. cloudCover/100 * moistureQM -> dewpointTemperatureQM * cloudAmount no units * updated bufr with new dim names * update prep. few names. add dimens * most updates for prep * No more 'Seen'. Other updates * delete metarsequence, Of not of * Move VSSO to be with other variables related to clouds * Rearrange the order of entries in the YAML * fix spacing in the YAML * Reorder entries in YAML * fix spaceing * Rearrange entries * Modify variable names and bug fix for ADPSFC BUFR and PREPBUFR YAMLS. Test output files also updated. * Update CMakelist.txt and add comments * Remove obselete files * Modify ADPSFC test output back to original * Modify variable names for max/min Temperatures * Add missing variables and rearranged entries. * Update test output for ADPSFC * Change heightOfStation to heightOfObservation * Update ctest output file for prepbufr adpsfc * get rid of v1 adpsfc. final recommended updates * revert, keep gdas.t06z.adpsfc.tm00.bufr_d for adpsfc_snow * re-add adpsfc_snow input bufr to CmakeLists.txt * Add the correct input test file for adpsfc snow data * update prepbufr file names, CMake, yaml obsdatain and out * typo * degrees_north and east -> degree_north and east * delete 'adsfc' file, update with degree Co-authored-by: Emily Liu <emily.liu@noaa.gov> Co-authored-by: Ron McLaren <ronlaren@aol.com> * Feature/query cxx atovs (#963) * Update test input files so data from all related satellite platform are included. Rearrange the entries in the YAML files. * update tes output files so that the naming convention of the filename is consistent. * Update test output for iodaconv fortran * Added code for automated type selection. * Removed check for unit type mismatch. * Fixed bug with string fields. * Missing values now depend directly on the types (always max value for the type). Removed Result struct in favor of DataObject as it was kinda redundant. * Updated DatetimeVariable.cpp code for optional value of Minute (#949) * Updated DatetimeVariable.cpp code for optional value of Minute * added a yaml file to test the DatetimeVariable.cpp code for optional MINU variable Co-authored-by: Emily Liu <emily.liu@noaa.gov> * Automatic type selection based on NCEPLIB-bufr refernece, scale and unit. * Added target.h and updated cmakelists.txt * updated mhs test file * Added way to manually override the type. * Print queries now displays type information. * Fixed issue with discovering if field is an integer. * Added some doc strings. Fixed readme. * Improved the handling of data types when doing transforms. * Fixed issue where transforms were being applied to missing value values. * Added support for the latest changes to NCEPLIB bufr * Updated calls to NCEPLIB-bufr * Now uses OOPS logging. * Changed things back to using a NCEPLIB function to get table b data instead of other API calls. * Updated for latest NCEPLIB-bufr enhancements branch. It removed get_tabb_info and added nemtbb and nemtab. * Fixed some comments * cast time to seconds instead of dividing by 1000 * fixed norm test failure * made exit into exception * using back inserter to append to sequence path * fixed error message. * fixed error message. * added prepbufr to git lfs * removed line continuation characters. * replaced 10e10 with missing value * accidentally deleted line of code. added it back * changed print time function back tot he original. * using gsl::narrow instead of simpoe static cast * Enhanced vector product function * Pulled out missing value into a Constants.h file to avoid duplication. * fixed some comments * fixed file permissions * Added try catch statements to main functions. Fixed warning. * pre reserve some memory * removed unused test from CMakeLists.txt file. * fixed norm check issues * Added include of unordered map as some compilers were not finding it. * Reverted unintended changes to files in ncar-bufr2nc-fortran and lib-python * Fixed bug that invovled parsing the bufr table data incorrectly. Basically we were not exiting trivial sequences properly when parsing the table. * Fixed bug in print queries as it was not trimming subset trims properly and was getting confused. * merged latest from query_cxx. fixed issue in print queries where it would crash trying to get type information in some instances. * Fixed some warnings and errors that happen with the CLANG compiler. * Enable the C++14 standard for C++ compiles. (#1036) * Fixed some code review issues that Cory Martin broght up. * updated docs for DataObject::print function * commented in line that was accidentally commented out * Revert "commented in line that was accidentally commented out" This reverts commit c78b93eae745b64b45567cec0ec2388ec129d0fb. * Added back missing tests from the develop branch. * Moved test_iodaconv_generic_gnssro_bufr back to where it was before (positionaly). * Update unit tests files and fixed some bugs so everything passes now. * Fixed issue caused by code review comment fix that caused bufr2ioda.x to obfuscate excepotion descriptions. * Fixed exception handler in print_queries.x. * Updated readme with a type example. * Pushing files onto git-lfs * Removed support for unsigned types as Fortran does not support them. * Updated adpsfc types (#1034) * Updated adpsfc types * Update test data files for adpsfc bufr and prepbufr Co-authored-by: Emily Liu <emily.liu@noaa.gov> Co-authored-by: Ron McLaren <ronlaren@aol.com> Co-authored-by: Emily Liu <emily.liu@noaa.gov> Co-authored-by: emilyhcliu <36091766+emilyhcliu@users.noreply.github.com> Co-authored-by: YoulongXia-NOAA <51668688+YoulongXia-NOAA@users.noreply.github.com> Co-authored-by: PraveenKumar-NOAA <63739712+PraveenKumar-NOAA@users.noreply.github.com> Co-authored-by: Nicholas.Esposito <Nicholas.Esposito@noaa.gov> Co-authored-by: haydenlj <99764299+haydenlj@users.noreply.github.com> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: AshleyStanfield-NOAA <92459593+AshleyStanfield-NOAA@users.noreply.github.com> Co-authored-by: NicholasEsposito-NOAA <62616739+nicholasesposito@users.noreply.github.com> Co-authored-by: Stephen Herbener <stephen.herbener@gmail.com> Co-authored-by: Cory Martin <cory.r.martin@noaa.gov> Co-authored-by: Benjamin Ruston <93737224+BenjaminRuston@users.noreply.github.com> * Feature/pace l1b converter (#1062) * added pace radiance converter * Update CMakeLists.txt * .. * .. * .. * modified to pass code norm * .. * .. * fix the help part * .. * .. * cleanup and changing the testoutput file Co-authored-by: Benjamin Ruston <93737224+BenjaminRuston@users.noreply.github.com> * Feature/query labeled dims (#1055) * First cut at calling the new nceplibs bufr query interface. * Updated to handle multi-dimensional data. Fixed DatetimeVariable. Got bufr mhs example to work. * Added support for CCITT IA5 string fields. Fixed many other fields that not not properly support queries. * Added support for forfield. Cleaned up some uneccesarrily complicated data structures. * Removed split map in favor of a list of splits to make code more readable. * Fixed bug with transforms. Updated some YAML files. * Updated the README file. * Fixed bug were CCITT IA5 fields would fail because of a silly mistake (throwing exception unneccessarliy). * Renamed for field to group_by field. * Removed duplicate directory for transforms. * incremental checkin, getting higher dimensional data working * Added DataObject cpp * Finished major code revisions and got things to compile for automatic dimensioning. * Some bug fixes to get simple radiance data working. * …
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
6 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Adds nchans = 4 to the viirs_aod2ioda.py converter.
This is needed for the newer verisions of fv3jedi_var.x, which will fail without the channel number.