Release/v0.8.0 #1138
Merged
Release/v0.8.0 #1138
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
* Move filter coefficients and decimation factor to variables * Fix bug with getting the wrong coeff values * Apply suggestions from code review Co-authored-by: Wu-Jung Lee <[email protected]> * Update to ignore empty data * Update test to check on ds not attrs * Clean up coeff retrieval to ensure both are checked --------- Co-authored-by: Wu-Jung Lee <[email protected]>
* Fix calibrate to grab variables not attributes * Return None when it doesn't exist at all
* Define backscatter_r and backscatter_i long_name in the convention yaml file, then use it consistently everywhere * Correct backscatter_r and backscatter_i units for AZFP (count, not dB) and EK80 (dB, not V in some cases) * Remove some non-operational # noqa flags added recently to set_groups_ modules
* Initialize combine_echodata overhaul * Remove print statement on test * Update multi combine test and fix empty prov dims * Modify group_paths to return tuple instead of set * Move globar var * Modify combine for provenance of attributes * Modify echodata test to be a tuple * Add more comments and docstrings * Remove unneeded functions * Remove extra filename setting in dataframe * Update echopype/echodata/combine.py Co-authored-by: Aniket Fadia <[email protected]> * Add vendor specific group checking * Update check_filter_params for group having ds_append_dims * Fix how encodings are set for data chunks * Apply suggestions from code review Co-authored-by: Wu-Jung Lee <[email protected]> * Apply suggestions from code review Co-authored-by: Wu-Jung Lee <[email protected]> * Rename echodatas * Remove reference to zarr_path * Fix ref to check_echodata_inputs * Add comments to _merge_attributes * Remove EK80 if statement * Move echopype group to attribute within provenance * Rename _check_filter_params and fix bugs appending multi combined * Remove checking preferred_chunks * Add chunk optimization during encoding determination * Add another line of docstring to test * Update attributes merging to store first value --------- Co-authored-by: Aniket Fadia <[email protected]> Co-authored-by: Wu-Jung Lee <[email protected]>
…sion (#1058) * Update convention yaml with corrected long_name entries and adding platform sentence_type * EK80 Platform updates: cosmetic changes for clarity and consistency; fixed or added missing long_name for two variables * EK60 Platform updates: ensured platform_* global attributes are inserted; added long_name to sentence_type * AZFP Platform updates: added three coordinates; added many missing variables, filled with nan, following dimensionality as found in EK80; cosmetic changes for clarity and consistency * Remove from AZFP Platform test check the variables that are no longer scalar * When azfp tilt_x and tilt_y are all nan, create single-valued arrays * Update echopype/convert/set_groups_azfp.py Co-authored-by: Wu-Jung Lee <[email protected]> * For EK60 Platform, removed incorrect channel dimension from several variables; set param_* global attributes to empty string (removed use of ui_param); and cosmetic tweaks * Removed time3 from Platform and removed it from all variables using it, driven by water_level (to adhere to convention where it's a scalar); set Platform platform_* global attrs to empty string * In AZFP Platform group, change tilt units to arc_degree and set time1 as a variable at the start for clarity * Update ek80 convert tests to reflect removal of Platform time3; and also that drop_keel_offset valid values are now handled correctly after replacing hasattr with dict .get in ek80 Platform set_group * Update echopype/tests/convert/test_convert_ek80.py --------- Co-authored-by: Wu-Jung Lee <[email protected]>
… [all tests ci] (#1061) * Fix processing level L1A test to account for AZFP open_raw now creating empty lat & lon variables * Remove obsolete environmental parameters tests * Fix test_plot tests and visualize api involving water_level > vertical_offset and the removal of channel dim from ek60 platform vars * Cosmetic change simply to trigger all ci tests * Remove remnant test case for ek60 vertical_offset channel dim; correct data-type typing hint
* Update convention yaml with corrected long_name entries and adding platform sentence_type * EK80 Platform updates: cosmetic changes for clarity and consistency; fixed or added missing long_name for two variables * EK60 Platform updates: ensured platform_* global attributes are inserted; added long_name to sentence_type * AZFP Platform updates: added three coordinates; added many missing variables, filled with nan, following dimensionality as found in EK80; cosmetic changes for clarity and consistency * Remove from AZFP Platform test check the variables that are no longer scalar * When azfp tilt_x and tilt_y are all nan, create single-valued arrays * Update echopype/convert/set_groups_azfp.py Co-authored-by: Wu-Jung Lee <[email protected]> * Overhauled echodata update_platform to support only a user-provided dictionary of variable mappings to the external data. Still messy code, and an issue has come up with deleted variables * For EK60 Platform, removed incorrect channel dimension from several variables; set param_* global attributes to empty string (removed use of ui_param); and cosmetic tweaks * Removed time3 from Platform and removed it from all variables using it, driven by water_level (to adhere to convention where it's a scalar); set Platform platform_* global attrs to empty string * In AZFP Platform group, change tilt units to arc_degree and set time1 as a variable at the start for clarity * Update ek80 convert tests to reflect removal of Platform time3; and also that drop_keel_offset valid values are now handled correctly after replacing hasattr with dict .get in ek80 Platform set_group * Overhaul echodata update_platform to operate off a user-provided dictionary of variable name mappings from Platform var names to external-data var names. Currently implements using time1 dim for all inserted variables * Remove stray comment block * Add missing check for lat-lon in class-method based processing level decorator. Also improve and reuse the general lat-lon test functionality * Overhaul update_platform to support user-specified dict of variable mappings with variables on potentially >1 time dims, or no time dim (scalar variables) * Accommodate more timeX dimension names in DEFAULT_ENCODINGS * Update the update_platform docstring * Update add_location to accommodate the flexibility in time dimension names now implemented in update_platform * Update tests that broke with the overhaul of update_platform * Update echopype/echodata/echodata.py Co-authored-by: Wu-Jung Lee <[email protected]> * Update echopype/echodata/echodata.py Co-authored-by: Wu-Jung Lee <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * Update echopype/echodata/echodata.py Co-authored-by: Wu-Jung Lee <[email protected]> * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * In update_platform: Changed mappings_expanded to a dict of dicts; added verifications for latitude and longitude; variable attribute assignments will now be made based on the pre-existing Platform variable's attributes * Add test_update_platform_multidim test to test platform updates with external data on two time dimensions plus an external scalar variable. Also small readability improvement to test_update_platform * add TODO for reviewing sorting external timestamps block * add test for lat/lon mismatches or only one exists * add warning when no variables are updated, add related test * revise docstring to include inspecting existing variables Co-authored-by: Emilio Mayorga <[email protected]> * fix string * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove empty declaration * add TODO for allowable platform variables --------- Co-authored-by: Wu-Jung Lee <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…roup (#1067) * add handling of problematic NMEA strings * fix bug re no lat/lon scenario * add warning for problematic lat/lon entry * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * fix dimension mismatches * remove conversion to numpy array for msg_type --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* Fix object dtype and cast to string * add cal_channel_id to EXPECTED_VAR_DTYPE in utils.coding * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove accidental test code --------- Co-authored-by: Wu-Jung Lee <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* change EK80 xml in vendor group to be variable instead of attribute * move azfp vendor attrs to variables * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove noqa * add back accidentally removed , --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* update mpl.cm.register_cmap with matplotlib.colormaps.register * pin matplotlib to >=3.7
* revise implementation and rename _clean_ping_time to _clean_reversed * add tests/qc/test_qc.py * add qc to test suite * update to use win_len median, update tests to use real datetime64[ns] values * add test_exist_reversed_time * fix typing for win_len Co-authored-by: Emilio Mayorga <[email protected]> * Apply suggestions from code review Co-authored-by: Emilio Mayorga <[email protected]> * fix missed param from suggested commits in test --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> Co-authored-by: Emilio Mayorga <[email protected]>
* Remove deprecated preprocess subpackage * docs: replace references to preprocess with clean and commongrid * docs: add attention box about removal of preprocess subpackage
* Remove beam dimension previously added during EK60 conversion * Handle case in EK calibration where beam dimension is not present * Remove beam dimension previously added during AZFP conversion * Remove handling of beam dimension in AZFP calibration * Update convert and echodata tests to account for elimination of beam dim from EK60 and AZFP * Forgot to undo in previous commit the temporary removal of netcdf4 test output in export_engine * Remove forced addition of beam dim in EK80 when not required. Now used only with complex samples * Remove forced handling of beam dim in EK80 calibration when beam is not present * Update convert, echodata and calibration tests to account for selective elimination of beam dim from EK80 * Forgot to undo in previous commit the temporary removal of netcdf4 test output in export_engine * fix test_nan_range_entries * fix test mock ata * remove added if-else * remove beam dim handling in calibrate/cal_params.py * revise comment re one place where the beam dim should be dropped explicitly * remove outdated comments re which var has beam dim * remove redundant check for if beam dim exists * remove beam dim from test_cal_params.py::beam_AZFP * remove beam dim check cal_params.py::_get_interp_da under the alternative case --------- Co-authored-by: Wu-Jung Lee <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* remove artifically added ping_time dim from set_groups_* * remove check of ping_time in split_beam_angle.py::get_angle_complex_samples * add additional comments to test_cal_params_integration.py * add additional comments to calibrate_ek.py::_assimilate_ecs_cal_params * remove unused (already commented out) lines in ek80_complex.py::compress_pulse * update cal_param.py::_get_interp_da on handling with some params now without ping_time dim * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove isel(ping_time=0) for beam_type split_beam_angle.py::get_angle_complex_samples * revise comments and fix typo --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* docs: remove outdated Attention box in Data Processing page * Add 3 non-convention variables to EK80 Beam groups; add missing long_name attr to slope var
Sync's up 'main' and 'dev' so that the 'dev' branch has the latest tags
* docs: remove outdated Attention box in Data Processing page * Remove count variable from EK60 Beam_group1, for consistency w/ EK80 and to remove redundancy * In EK60 Beam_group1, rename offset to range_sample_offset, cast it and data_type to int types, and improve the attributes. * Rename ek60 transmit_mode to channel_mode (per ek80) and harmonize attributes. Ensure int casting works as intended via encoding
* ci: Update PR GH Workflow * ci: Fix PR Workflow config * ci: Fix matrix config for PR workflow config * ci: Fix runs-on values for PR workflow config * ci: Fix py3.11 matrix config * ci: Remove macos config since docker not supported * ci: Update configs for build workflow and py3.11 * ci: Remove build py3.11 * ci: Fix path to test log config
…tributes (#1107) * Change AZFP data types for backscatter_r and gain_correction to float, to match convention * Add long_name attribute for many AZFP Vendor variables * Add long_name and units attributes for a couple more AZFP Vendor variables
* Add missing, mandatory Beam_group beam_stabilisation and non_quantitative_processing, as scalars * Add mandatory Beam_groupX sample_time_offset to replace the now redundant (for EK60/80) range_sample_offset
…fe (#1110) * [pre-commit.ci] pre-commit autoupdate (#1109) updates: - [github.com/PyCQA/flake8: 6.0.0 → 6.1.0](PyCQA/flake8@6.0.0...6.1.0) Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com> * Changed test file name to UUID * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
…rroring conversion_* attributes (#1113)
* AZFP: add mandatory beam_type and transmit_type * AZFP, add channel dimension to beam_type for consistency with EK60 & EK80 * Add beam_direction_x/y/z Beam_group1 variables to AZFP, setting them to nan
…m` for BB checks (#1091) * docs: remove outdated Attention box in Data Processing page * In EK80, rename frequency_start/end to convention-based transmit_frequency_start/stop * For EK80 transmit_frequency_start/stop, standardize data type (int to float) and attributes * Add transmit_frequency_start/stop to EK60 & AZFP, set to frequency_nominal * Replace use of frequency_start/stop in ek80 checks for BB channels with pulse_form * Rename pulse_form to transmit_type per convention, and map int codes to strings (CW, FM, FMD) * Assigned attributes to the wrong variable, in previous commit * The meaning for ek80 channel_mode = 1 is not described in ek80 manual * Ensure variables remain encoded as int types even after xr.merge * Overhaul EK80 checks for BB vs CW to use pulse_form / transmit_type * Set transmit_frequency_start/stop attributes in 1.0.yml and update set_groups_azfp/ek60 to use that * For ek80 transmit_freq_start/stop is now created and/or populated with frequency_nominal for power-only channels. Updated all BB vs CW checks. * Simplified ek80 tapered_chirp now that CW channels always contain freq nominal in transmit_frequency_start/stop * Small cleanups to comments, unused function parameter, etc, all involving ek80 * Simplify ek80 test for encode_mode = 'complex' but there are no CW pings * Remove transmit_type from tx_param_names b/c it's no longer an argument in tapered_chirp * Optimize ek80 BB vs CW tests using transmit_type on first ping_time * Update echopype/echodata/simrad.py Co-authored-by: Wu-Jung Lee <[email protected]> * Update echopype/echodata/simrad.py Co-authored-by: Wu-Jung Lee <[email protected]> * Change transmit_type FM string to LFM, per convention, and add more detailed flag_meanings strings, largely from the convention --------- Co-authored-by: Wu-Jung Lee <[email protected]>
…n_x/y/z to nan as needed (#1114) * Add sample_time_offset to EK60; looks like I missed it earlier * Set beam_direction_x/y/z to nan for ek60 & ek80 when all 3 values are 0
* fix: fix how provenance data are combined * test: add test to check actual prov vals * fix: remove ECHODATA_FILENAME, use ED_FILENAME * tests: simplify 'test_combine_echodata_combined_append' (lsetiawan#1) Simplified by using common code for repeated blocks; plus other readability tweaks --------- Co-authored-by: Emilio Mayorga <[email protected]> Refs: #1115
…ombine & duplicate / old ping_time stuff [all tests ci] (#1117) * Set ping_time arrays explicitly to datetime64[ns] in parsers, to silence new run-time warning * Set ek datagram timestamp arrays explicitly to datetime64[ns], to silence new run-time warning * fix selected ping_time getting tested * fix _print_status casting * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * update test_vertical_offset_echodata and test_vertical_offset_Sv_dataset to isel for ping_time instead of actual timestamp values * Remove test_convert_ek60_duplicate_ping_times as changing ping_time resolution to ns eliminated the previous identification of duplicates * Remove obsolete ZarrCombine and associated tests * Remove EK60 duplicate ping_time functionality, including old_ping_time Provenance variable * A custom SetGroupsEK60 set_provenance method is no longer needed --------- Co-authored-by: Wu-Jung Lee <[email protected]> Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* remove 1-element handling for wbt_fil/pc_fil due to format change * factor out normalization original in ek80_complex.compress_pulse into new function ek80_complex.get_norm_fac * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * add temp changes and mark potential BUG * add check for angle_param in source_Sv, in consolidate.add_splitbeam_angle * remove test filepath * use Literal typing for filter_name in get_vend_filter_EK80 * add test_ek80_complex.py::test_get_vend_filter_EK80 * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* re-render ek60 echodata example, add ek80 and azfp examples * small wording changes * add echopype version * re-run notebook * tweak text and sections * fix echopype version print out
* re-render ek60 echodata example, add ek80 and azfp examples * small wording changes * add echopype version * update power/angle data beam dim description * add consolidate to api.rst * try to link add_splitbeam_angle * try to link consolidate.add_splitbeam_angle * start overhaul data proc page in new notebook, bibtext does not work * add cal_params and env_params sections * remove old data processing page * add interfacing ecs file * re-run notebook * tweak text and sections * fix echopype version print out * add metrics subpackage into docs * fix: process subpackage is now removed * separate out section for sonarnetcdf4 adaptation, add nan-padding fig * actually add adaptation page, update toc * add image for echo_range and revise text * add allowable cal params for each instrument type * add docs/images * reorganize data processing pages * fix typo * push up new sets of data-proc files
* docs: Correct spelling error * docs: Fix RTD latex rendering error * docs: move 'Format of processed data' section to more prominenbt page * update range_bin to range_sample in fig --------- Co-authored-by: Wu-Jung Lee <[email protected]>
) * docs: Correct spelling error * docs: Fix RTD latex rendering error * docs: move 'Format of processed data' section to more prominenbt page * docs: Add section on z-axis vertical coordinate system * docs: Add z-axis vertical coordinate section * edit z-axis docs: remove some repetition and fix beam_direction_* text * add z-axis vars diagram * update z-axis vars diagram * update z axis vars fig * Update docs/source/data-proc-additional.ipynb Co-authored-by: Wu-Jung Lee <[email protected]> * fix sample_depth in diagram * docs: shift up the occurrence of the vertical variables figure * docs: add brief admonition note about add_depth; rename sample_depth references to depth * update fig: sample_depth to depth --------- Co-authored-by: Wu-Jung Lee <[email protected]>
* docs: update SONAR-netCDF4 v1 url * docs: fix open_raw api link * docs: Generalize the name and title of data-format-5to6 page * docs: Add section describing 0.7.1 to 0.8.0 major changes * docs: Add stub that will include brief reference to the echopype-checker package and repo * docs: add section on echopype-checker * docs: remove obsolete process.rst * tweak wording for v0.8.0 changes section * change v1 to version 1 --------- Co-authored-by: Wu-Jung Lee <[email protected]>
* comment out nasc-related sections * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci * remove import * fix pre commit problems --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* fix cite rendering problem * fix bib file * add bibliography file * [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --------- Co-authored-by: pre-commit-ci[bot] <66853113+pre-commit-ci[bot]@users.noreply.github.com>
* first draft v0.7.2 release notes * add update_platform entry * add notes on new PRs * add new changes * add recent PRs * update release notes * revise release date * add new changes * add #1137 entry
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
Codecov Report
@@ Coverage Diff @@
## main #1138 +/- ##
==========================================
- Coverage 80.71% 79.78% -0.93%
==========================================
Files 67 65 -2
Lines 6056 6011 -45
==========================================
- Hits 4888 4796 -92
- Misses 1168 1215 +47
Flags with carried forward coverage won't be shown. Click here to find out more.
... and 2 files with indirect coverage changes 📣 We’re building smart automated test selection to slash your CI/CD build times. Learn more |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR contains all changes in release v0.8.0.