Feature/query datetime subset fix#1059
Merged
BenjaminRuston merged 12 commits intofeature/query_labeled_dimsfrom Sep 28, 2022
Merged
Feature/query datetime subset fix#1059BenjaminRuston merged 12 commits intofeature/query_labeled_dimsfrom
BenjaminRuston merged 12 commits intofeature/query_labeled_dimsfrom
Conversation
…n the query subset defenition.
rmclaren
requested review from
CoryMartin-NOAA,
emilyhcliu,
nicholasesposito and
srherbener
| /// \brief Get the data at the index as an int. | ||
| /// \return Float data. | ||
| virtual float getAsFloat(size_t idx) const = 0; | ||
| virtual int getAsInt(size_t idx) const = 0; |
Contributor
There was a problem hiding this comment.
Ron, I am using the getAsFloat function in my feature/query_cxx_types_NE_aircraftAltitude branch. I think we should keep both
Contributor
Author
There was a problem hiding this comment.
Actually I really need to get rid of both as using this function is really dangerous if you are not careful... If you depend on any values in repeated sequences (or that could be in repeated sequences) you should not go near these methods. I'm gonna have to think about this a little I think.
Contributor
There was a problem hiding this comment.
OK, I'll rework my code to not use it.
Contributor
Author
There was a problem hiding this comment.
Ok, I added it back. You need to validate the input data and make sure it all comes from the same place (same place I the tree).
Contributor
Author
There was a problem hiding this comment.
This will need to be revisited in the future.
nicholasesposito
approved these changes
Sep 23, 2022
Contributor
nicholasesposito
left a comment
nicholasesposito
left a comment
There was a problem hiding this comment.
Lots of good changes in this update, especially regarding the determination of array sizes. I will definitely use this to fix up my own aircraftAltitude. Thanks for doing this.
BenjaminRuston
added
the
waiting for another PR
BenjaminRuston
added a commit
that referenced
this pull request
Sep 29, 2022
* First cut at calling the new nceplibs bufr query interface. * Updated to handle multi-dimensional data. Fixed DatetimeVariable. Got bufr mhs example to work. * Added support for CCITT IA5 string fields. Fixed many other fields that not not properly support queries. * Added support for forfield. Cleaned up some uneccesarrily complicated data structures. * Removed split map in favor of a list of splits to make code more readable. * Fixed bug with transforms. Updated some YAML files. * Updated the README file. * Fixed bug were CCITT IA5 fields would fail because of a silly mistake (throwing exception unneccessarliy). * Renamed for field to group_by field. * Removed duplicate directory for transforms. * incremental checkin, getting higher dimensional data working * Added DataObject cpp * Finished major code revisions and got things to compile for automatic dimensioning. * Some bug fixes to get simple radiance data working. * Enhanced the reading of th dimensions field. * Got filtering and splitting to work again. Might need to be a little more careful with slice dimensions, but works with existing examples. * Fixed a dimension naming issue with the way group_by location dimensions were being named. Fixed some compiler warnings. * Added arguments to bufr2ioda so you can limit the amount of messages that are parsed. * Changed dimensions attribute to list or single string. * Fixed transforms. * Fixed transfoms so they will not affect missing values. * Fixed bug that happened when we tried to export an empty category. * Fixed strange issue that happened while applying filters. Removed some redundant yaml files from testinput. * Updated a bunch of unit tests so that they would work with the latest code. * updated test output files to work with latest version of ioda * Added unit test and now time the runtime. * Fixed a couple more unit tests. * Added .dump files to gitlfs * removed uneeded file * BUFR2IODA conversion for AMSU-A and MHS Radiance data (#713) * Add bufr to ioda conversion for IASI, ERAS/DB AMSU-A and MHS for ncar-bufr2nc-fortarn utility * Add tests and associated test input/output files for ncar-bufr2nc-fortarn utility * Add NCEP BUFR tables for normal-feed and RARS/DB satellite radiance data * Add YAML files, tests, and test reference for normal-feed and db/ears AMSU-A * Fix coding norm * Rename NCEP BUFR tables for satellite radiance data * Add yaml files, tests, and test input/output files for MHS (normal-feed and RARS/DB) t * Update Split.cpp * Update DataObject.cpp * Update IodaEncoder.cpp * Update bufr_ncep_1bamua.yaml * Update bufr_ncep_1bmhs.yaml * Update bufr_ncep_esamua.yaml * Update bufr_ncep_esmhs.yaml * Fixed coding norms issue. * Fixed some little issues. Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Update README.md Updated documentation for dimensions. * Test and enhance two yaml files for adpsfc snow and sevcrs data (#736) * Update snowadpsfc * Update bufr_snow_adpsfc.yaml * Update testoutput * Update yaml files and testinput and testoutput data * Reduce input data size to 20% and update output * Reduce input data size to 250KB and update testoutput * Reduce the size of test files for amsua, mhs and iasi. Update the corresponding test output files (#764) * Removed uneeded files I missed during merge. * Initial checkin of query classes with a lot of placeholder code * Got data collection mostly working. * Starting to work end to end. Need to fix exportdims and revisit code that deals with group by fields. * Fixed issue where targets and masks were being copied unecessarily. * Feature/adpupa prepbufr (#794) * Added adpupa prepbufr yamls w/o group_by field with input adpupa prepbufr file * adpupa netCDF files w/o group_by field * Updates made after ioda-validate.x run. * Changed Quality Mark group from "ObsQualityMark" to "QualityMarker" * Got a bunch of unit tests working. * add aircft profiles to feature/query * Fixed non-working functional unit tests. * Added unit test for simple groupby example. * Made simple groupby test work. Fixed related bugs. * Added unit test for bufr files with missing fields. * Added unit test for sfcshp data * accidentaly messed up the bufr-mhs test. this fixes it * Integrated latest bug fixes from the feature/query branch. * Fixed bug were we weren't checking for empty sequence counts. * added adpupa tests * Fixed bugs that prevented adpupa from working. * Getting much closer to having adpupa groupby example working. * fixed final bug to get adpupa groupby case to work the same way. * Removed some uneeded code. * Reduce the size of the ADPUPA test input furrther * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. * Removed undesired include from legacy bufr code. Fixed some copyright statements. * Changed the way data provider works (no longer a singleton). Minor cleanup. * Added print_queries tool to the ioda-converter project. * Improved situation arrising from missing string fields. * Improved detection of string fields. * Moved table logic out of the print_quiries function. Fixed bug were idxs were not being displayed on queries. * Replace nlocs with Location, reduce the size of ADPUPA.prepbufr and … (#884) * Replace nlocs with Location. Reduce the size of ADPUPA.prepbufr and update the test output. * remove checking of satellite id in QC (#878) * remove checking of satellite id in QC * added checking of bits 1 and 4 in QC flag Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> * Feature/optional python qc (#881) * Corrected bit checks due to reverse ordered bits * add check if there any data to write out * update ctest input and reference file * add new test output verification * corrected pycodestyle error * added optional georeality QC checks similar to those present in FORTRAN converter * pycodestyle corrections * removed percent confidence check * attempting fix of conflicts with existing code * corrected syntax errors * removed global variable Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: haydenlj <99764299+haydenlj@users.noreply.github.com> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Reverted ADPUPA bufr file to smaller one. * Fixed compiler warnings * Fixed code style (norm) issues. * Added doc strings to query code. * Feature/query cxx praveen (#893) * adpupa prepbufr yaml for updated group_by field and corresponding testoutput file * adpupa prepbufr yaml containing GSI mnemonics only and corresponding output file * adpupa gsi prepbufr file with updated name, groups, units, with bufr2ioda_cxx.x generated netCDF file * added TVO and CLTP to the adpupa prepbufr yaml, updated name and output netcdf files * Updated dimensions into yaml * Added group_by fields for ELV, TYP, and T29; and updated the nc files * Added white space * Change dimension name to following unified IODA naming convention (#767) * Reduce the size of test files for amsua, mhs and iasi. Update the corresponding test output files * Replace "nlocs" with "Location" and "nchans" with "Channel" (to following the unified IODA naming convention). * Fix a typo and update test output accordingly * Added white space * Made updates to make everything consistent with query_cxx branch Co-authored-by: Ronald McLaren <ronald.mclaren@noaa.gov> * Fixed bug when there were more than one subset in the file. Added ability to print monotone output (so you can pipe to a text file). * Updated print queries so it no longer uses colors but prints the dimensioning subpaths seperatly. * Chenged some maps to unordered_maps as they are better performance wise. * Feature/query cxx as (#914) * Updated yaml, yamlinput, and yaml output for SatWind AVHRR NC005080 within the new feature/query_cxx branch. * The bufr_ncep_sawtind_avhrr yaml has been updated to include and produce input and output with the same name scheme. Files have been renamed accordingly. * Modified input and output fields of the yaml file. * modified: CMakeLists.txt renamed and modified: bufr_ncep_satwind_avhrr.yaml -> testinput/bufr_ncep_satwind_avhrr.yaml The input fields and linked name fields for "time" has been updated as suggested. The CMakeList has been updated to match with the new naming scheme. The "dimensions" field has been updated as suggested. And the yaml has now been moved from ./test to ./test/testinput. * Removed indices for most variables, while adding in 2 more indices for GNAP and PCCF to get wind direction and wind speed generating application and percent confidence. * Updated variables names to match the JEDI convention spreadsheet. * Removed a default/backup configuration line. * Modify unit attribute in BUFR2IODA converter and bug fix for satellite winds (#940) * Make units attribute an optional variable * Bug fix for AVHRR satellite wind test: (1) unit in yaml (2) output directory in yaml (3) test output file in CMakelist.txt * Change the test input and output file names according to the following logic: (1) for input, use the global dump file naming convention (e.g. gdas.t00z.satwnd.tm00.bufr_d) (2) for output change the input file name extension bufr_d to nc (gdas.t00z.satwnd.tm00.nc) * delete bufr_ncep_satwind_avhrr.nc * Make sure the time info is consistent between the file name and the data in the file. * adjust the order of files listed in the CMakeList.txt * Change the way to test the empty string. * Fixed issue where non-existing query with index would cause a crash. Also fixed bug where the suggested dimesioning paths would be messed up sometimes. * Fixed some norm test failures. * Fixed all the unit tests * Made change so that there is always one group_by field whcih applies to all the variables. * Removed uneeded test * Improved some of the code to do with finding the data object to group against. * Fixed bug in print queries were fixed repeats were being missed. * update two yaml files via query_cxx (#906) * update snow adpsfc yaml and the result * Update sevcsr yaml file and the result * Update bufr_ncep_read_2_dim_blocks.yaml * Fix dateTim with DateTime, nchans with Channel * Remove ncep for yaml and modify the CMakeLists.txt * Update bufr_ncep_snow_adpsfc.yaml * Update bufr_read_2_dim_blocks.yaml to follow JEDI convention and update the associated test output (bufr_read_2_dim_blocks.nc). * Modified yaml, add sacle and update the result * Changes cloudAmount unit and update the result * Change cloudAmount to cloudCoverTotal * Set BUFR variable: CLDMNT to cloudAmount since it is already in JEDI convention with optional dimenion as [location, channel] Co-authored-by: Emily Liu <emily.liu@noaa.gov> * Feature/query cxx mhs (#960) * Update 1bmhs YAML and test output * Update DB mhs YAML file and test output * Modify longnames and update test output files * Feature/query cxx amsua (#958) * Update YAMLS and test output files for 1bamsua and esamsua * Update YAML files for AMSU-A * Remove unwanted test file for amsua-a * fix spacing. * add spacings between lines * Change height to heightOfStation and update test output * Fix longnames * Feature/query cxx iasi (#966) * Update IASI yaml and related testoutput * Update IASI YAML * Update IASI test file * Fix a few errors in IASI YAML and update the test output * Feature/query cxx nick e adpsfc (#904) * adpsfc prepbufr and bufr * recommended changes * recommended updates 2 (* and deg) * CMakelists.txt update * update CMakeLists.txt * adpsfc prepbufr recommended changes * remove nc, add full adpsfcyaml * adpsfc updates, plus CMakelists * remove CLTP and VSSO * decrease number of variables * update output * heightOfStation -> stationElevation * get rid of oceanographic * update heightofbaseofcloud description * update with correct files * updates to HOCB CLAM VSSO * update thinned, has all subsets * Fixed the unit tests * reportid -> SID, CLAM[1] is CLAM, pW@Obs, percent ->1 * update output * new output from updated bufr2ioda.x * bufr: HSMSL+SELV=stationElevation, PB: percent->1 * get rid of units:"" and unitless. * add dimensions. cloudAmountObserved. cloudCover/100 * moistureQM -> dewpointTemperatureQM * cloudAmount no units * updated bufr with new dim names * update prep. few names. add dimens * most updates for prep * No more 'Seen'. Other updates * delete metarsequence, Of not of * Move VSSO to be with other variables related to clouds * Rearrange the order of entries in the YAML * fix spacing in the YAML * Reorder entries in YAML * fix spaceing * Rearrange entries * Modify variable names and bug fix for ADPSFC BUFR and PREPBUFR YAMLS. Test output files also updated. * Update CMakelist.txt and add comments * Remove obselete files * Modify ADPSFC test output back to original * Modify variable names for max/min Temperatures * Add missing variables and rearranged entries. * Update test output for ADPSFC * Change heightOfStation to heightOfObservation * Update ctest output file for prepbufr adpsfc * get rid of v1 adpsfc. final recommended updates * revert, keep gdas.t06z.adpsfc.tm00.bufr_d for adpsfc_snow * re-add adpsfc_snow input bufr to CmakeLists.txt * Add the correct input test file for adpsfc snow data * update prepbufr file names, CMake, yaml obsdatain and out * typo * degrees_north and east -> degree_north and east * delete 'adsfc' file, update with degree Co-authored-by: Emily Liu <emily.liu@noaa.gov> Co-authored-by: Ron McLaren <ronlaren@aol.com> * Feature/query cxx atovs (#963) * Update test input files so data from all related satellite platform are included. Rearrange the entries in the YAML files. * update tes output files so that the naming convention of the filename is consistent. * Update test output for iodaconv fortran * Added code for automated type selection. * Removed check for unit type mismatch. * Fixed bug with string fields. * Missing values now depend directly on the types (always max value for the type). Removed Result struct in favor of DataObject as it was kinda redundant. * Updated DatetimeVariable.cpp code for optional value of Minute (#949) * Updated DatetimeVariable.cpp code for optional value of Minute * added a yaml file to test the DatetimeVariable.cpp code for optional MINU variable Co-authored-by: Emily Liu <emily.liu@noaa.gov> * Automatic type selection based on NCEPLIB-bufr refernece, scale and unit. * Added target.h and updated cmakelists.txt * updated mhs test file * Added way to manually override the type. * Print queries now displays type information. * Fixed issue with discovering if field is an integer. * Added some doc strings. Fixed readme. * Improved the handling of data types when doing transforms. * Fixed issue where transforms were being applied to missing value values. * Got it kinda working. * slight improvemetn * slight improvement * Added support for the latest changes to NCEPLIB bufr * Updated calls to NCEPLIB-bufr * Now uses OOPS logging. * Changed things back to using a NCEPLIB function to get table b data instead of other API calls. * Updated for latest NCEPLIB-bufr enhancements branch. It removed get_tabb_info and added nemtbb and nemtab. * Fixed some comments * cast time to seconds instead of dividing by 1000 * fixed norm test failure * made exit into exception * using back inserter to append to sequence path * fixed error message. * fixed error message. * added prepbufr to git lfs * removed line continuation characters. * replaced 10e10 with missing value * accidentally deleted line of code. added it back * changed print time function back tot he original. * using gsl::narrow instead of simpoe static cast * Enhanced vector product function * Pulled out missing value into a Constants.h file to avoid duplication. * fixed some comments * fixed file permissions * Added try catch statements to main functions. Fixed warning. * pre reserve some memory * removed unused test from CMakeLists.txt file. * fixed norm check issues * Added include of unordered map as some compilers were not finding it. * Reverted unintended changes to files in ncar-bufr2nc-fortran and lib-python * Fixed bug that invovled parsing the bufr table data incorrectly. Basically we were not exiting trivial sequences properly when parsing the table. * Fixed bug in print queries as it was not trimming subset trims properly and was getting confused. * merged latest from query_cxx. fixed issue in print queries where it would crash trying to get type information in some instances. * Fixed some warnings and errors that happen with the CLANG compiler. * Enable the C++14 standard for C++ compiles. (#1036) * Fixed some code review issues that Cory Martin broght up. * updated docs for DataObject::print function * commented in line that was accidentally commented out * Revert "commented in line that was accidentally commented out" This reverts commit c78b93e. * Added back missing tests from the develop branch. * Moved test_iodaconv_generic_gnssro_bufr back to where it was before (positionaly). * Fixed failing unit tests * updated mhs unit test with dimension source to test this feature. * Moved applicable create dimension functions to be private scope. * Update unit tests files and fixed some bugs so everything passes now. * Fixed issue caused by code review comment fix that caused bufr2ioda.x to obfuscate excepotion descriptions. * Fixed exception handler in print_queries.x. * Updated readme with a type example. * fixed test * Updated code to take advantage of types branch * Updated docs strings for createDimension functions * Updated a unit test to check that dimesnion sourcing code is working. * Added some doc strings to ioda encoder. * Pushing files onto git-lfs * Removed support for unsigned types as Fortran does not support them. * pushing files into lfs * Added some validation for the dimension source fields. * Enhanced some doc strings * Fixed norm test issue * Updated the readme. * Updated adpsfc types (#1034) * Updated adpsfc types * Update test data files for adpsfc bufr and prepbufr Co-authored-by: Emily Liu <emily.liu@noaa.gov> Co-authored-by: Ron McLaren <ronlaren@aol.com> * Fixed some minor code review comments. * Feature/query datetime subset fix (#1059) * Fixed some bugs with the datetime object. Added some validation. * Made modifications so that the the messages would be filtered based on the query subset defenition. * Added configuration parameter called subsets to explicitly limit the valid subsets. * Improved the implemention a little. * Fixed coding norm issues * Renamed query function to accumulate. * Added some yaml files for testing purposes (still need to make the test data). * Added unit tests * updated readme * added getAsFloat back. * Fixed indentation in CMakeLists.txt file Co-authored-by: Emily Liu <emily.liu@noaa.gov> Co-authored-by: emilyhcliu <36091766+emilyhcliu@users.noreply.github.com> Co-authored-by: YoulongXia-NOAA <51668688+YoulongXia-NOAA@users.noreply.github.com> Co-authored-by: PraveenKumar-NOAA <63739712+PraveenKumar-NOAA@users.noreply.github.com> Co-authored-by: Nicholas.Esposito <Nicholas.Esposito@noaa.gov> Co-authored-by: haydenlj <99764299+haydenlj@users.noreply.github.com> Co-authored-by: Stephen Herbener <32968781+srherbener@users.noreply.github.com> Co-authored-by: BenjaminRuston <benr@ucar.edu> Co-authored-by: AshleyStanfield-NOAA <92459593+AshleyStanfield-NOAA@users.noreply.github.com> Co-authored-by: NicholasEsposito-NOAA <62616739+nicholasesposito@users.noreply.github.com> Co-authored-by: Stephen Herbener <stephen.herbener@gmail.com> Co-authored-by: Cory Martin <cory.r.martin@noaa.gov> Co-authored-by: Benjamin Ruston <93737224+BenjaminRuston@users.noreply.github.com>
emilyhcliu
removed
the
waiting for another PR
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Description
This pull request does several things:
Issue(s) addressed
Acceptance Criteria (Definition of Done)
Dependencies