Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add/icephys meta #1349

Merged
merged 119 commits into from
Jul 30, 2021
Merged

Add/icephys meta #1349

merged 119 commits into from
Jul 30, 2021

Conversation

oruebel
Copy link
Contributor

@oruebel oruebel commented Apr 5, 2021

Motivation

This PR is for integration of the ndx-icephys-meta structures with NWB core. Related PRs on the other repos are:

NeurodataWithoutBorders/nwb-schema#470
hdmf-dev/hdmf-common-schema#57
hdmf-dev/hdmf#551
oruebel/ndx-icephys-meta#82

Changes on this PR include

  • Declare SweepTable as deprecated on init
  • Integrated the new types from the ndx-icephys-meta extensions schema
    • TimeSeriesReferenceVectorData
    • IntracellularRecordingsTable
    • SimultaneousRecordingsTable
    • SequentialRecordingsTable
    • RepetitionsTable
    • ExperimentalConditionsTable
    • Port changes from ICEphysFile to NWBFile itself
  • Add new tutorial for the ICEPhys data structures and declare previous tutorial using SweepTable as deprecated
  • Add icephys functions to interact with pandas #1383

Current status update on the integration of the ndx-icephys-meta with the NWB core:

Checklist

  • Have you checked our Contributing document?
  • Have you ensured the PR description clearly describes problem and the solution?
  • Is your contribution compliant with our coding style? This can be checked running flake8 from the source directory.
  • Have you checked to ensure that there aren't other open Pull Requests for the same change?
  • Have you included the relevant issue number using #XXX notation where XXX is the issue number?

@oruebel oruebel requested review from rly and ajtritt April 5, 2021 19:25
@codecov
Copy link

codecov bot commented Apr 5, 2021

Codecov Report

Merging #1349 (fe4ce3d) into rc/2.0.0 (19ccf2f) will increase coverage by 3.70%.
The diff coverage is 92.55%.

Impacted file tree graph

@@             Coverage Diff              @@
##           rc/2.0.0    #1349      +/-   ##
============================================
+ Coverage     73.34%   77.04%   +3.70%     
============================================
  Files            37       37              
  Lines          2337     2692     +355     
  Branches        360      445      +85     
============================================
+ Hits           1714     2074     +360     
+ Misses          562      541      -21     
- Partials         61       77      +16     
Impacted Files Coverage Δ
src/pynwb/icephys.py 89.52% <88.05%> (-0.56%) ⬇️
src/pynwb/file.py 86.36% <96.59%> (+3.60%) ⬆️
src/pynwb/base.py 94.63% <98.50%> (+1.77%) ⬆️
src/pynwb/io/file.py 82.27% <100.00%> (+1.88%) ⬆️
src/pynwb/io/icephys.py 100.00% <100.00%> (+33.33%) ⬆️
src/pynwb/io/base.py 62.71% <0.00%> (+32.20%) ⬆️

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 19ccf2f...fe4ce3d. Read the comment docs.

@oruebel oruebel marked this pull request as ready for review April 13, 2021 16:41
@oruebel
Copy link
Contributor Author

oruebel commented Apr 13, 2021

@rly @ajtritt this PR is also ready for review. We should probably wait with the merge until NeurodataWithoutBorders/nwb-schema#470 and hdmf-dev/hdmf#551 have been merged so all the CI can be run. But other than that this PR should be complete.

src/pynwb/base.py Outdated Show resolved Hide resolved
src/pynwb/base.py Outdated Show resolved Hide resolved
src/pynwb/base.py Outdated Show resolved Hide resolved
src/pynwb/base.py Outdated Show resolved Hide resolved
src/pynwb/base.py Outdated Show resolved Hide resolved
src/pynwb/base.py Outdated Show resolved Hide resolved
src/pynwb/base.py Outdated Show resolved Hide resolved
src/pynwb/base.py Outdated Show resolved Hide resolved
tests/unit/test_base.py Outdated Show resolved Hide resolved
@rly
Copy link
Contributor

rly commented Jul 30, 2021

This is good to merge into rc/2.0.0. Great work!

@oruebel oruebel merged commit b5732b2 into rc/2.0.0 Jul 30, 2021
@oruebel oruebel deleted the add/icephys_meta branch July 30, 2021 15:55
rly pushed a commit that referenced this pull request Aug 10, 2021
Declare SweepTable as deprecated on init

Add TimeSeriesReferenceVectorData Container class

Add simple test for TimeSeriesReferenceVectorData

Update nwb-schema branch

Add icephys metadata container classes

Update NWBFile to support icephys metadata tables

Add unit tests for icephys metadata tables

Update imports and fix SweepTable integration tests

Fix test available namespaces testcase

Update SweepTable tutorial and declare as deprecated

Add new icephys tutorial using the new hierarchical metadata tables

Add IntracellularRecordingsTable illustration

Fix flake8 issues

Remove use of HierarchiclaDynamicTableMixin use for now

Add source PowerPoint file for icephys docs

add doc for TimeIntervals (#1352)

* add doc for TimeIntervals

* add allensdk back

Fix make clean command for docs to also clean sphinx-gallery files

Updated CHANGELOG to describe changes from this PR

Add thumbnails for domain-specfic tutorials

Add thumbnaisl for general tutorials

Update changelog

Remove ic_filtering/icephys_filtering duplicate and fix test

Fix description of ExperimentalConditionsTable

Use HDMF 2.5.0 (#1354)

DOC: make template less puzzling in how to fill out ;-) (#1336)

Co-authored-by: Ryan Ly <[email protected]>

Add ReadTheDocs badge to README (#1311)

Co-authored-by: Ryan Ly <[email protected]>

Update to use upcoming HDMF 2.5.1 (#1355)

Revert breaking changes to scratch API (#1356)

Use HDMF 2.5.2 (#1360)

Add `DfOverF` to docs (#1351)

Co-authored-by: Ryan Ly <[email protected]>

docs/gallery/general/file.py: Fix argument count in documentation

add strain optional arg to Subject (#1241)

* remove validate_core_schema.

You can check the core schema with e.g.

* Update Subject unit test

* Fix Subject unit test, add check for date_of_birth

Co-authored-by: Ryan Ly <[email protected]>

add source channels to DecompSeries

add option to include source_channels for DecompositionSeries (#1258)

* add option to include source_channels for DecompositionSeries and include round trip test
* Use latest schema
* Add unit test

Add optional link to Device in ImageSeries (#1265)

* Add optional link to Device in ImageSeries

* Add support for device in ImageSeries subclasses

* Update ImageMaskSeries docstring

* Update schema

add continuity optional arg to TimeSeries.__init__ (#1226)

* add continuity optional arg to TimeSeries.__init__

* use new enum feature of hdmf to check against controlled vocabulary

* Fix flake8

* Add integration test

* Fix flake8

* Add correct mapping for data.continuity

Co-authored-by: Ryan Ly <[email protected]>

Add support for ElectricalSeries.filtering dataset (#1270)

Set fixed value for IZeroClampSeries stimulus description

Fix test

Update changelog

Add note about allensdk py3.8 incompatibility

Add workaround for validation against cached hdmf-experimental

Update CI to avoid click8 incompatibility

Use HDMF 2.5.5

Address comments, improve doc

Fix docs, add citation info, update repo infra (#1362)

Use latest HDMF, raise min pandas, dissociate req-min and setup reqs (#1363)

Use nwb schema branch for icephys meta

Ensure start_index and count are set to -1 if TimeSeries is None

Add test for conversion of start_index and index_count vals

Fix flake8 in __compute_index docstring

Fix icephys metadata roundtrip NWBFile test base on update DynamicTableRegion behavior

Update schema (updated version to 1.4.0-alpha)

improve extensions documentation (#1350)

* improve extensions documentation

* adjustments to extensions.rst

* add tips to extension docs

* create multi-part extensions tutorial

* move API building docs to chapter 5 of extensions guide

* restructure and fix up some spots

* Update docs/source/extensions_tutorial/2_create_extension_spec_walkthrough.rst

Co-authored-by: Oliver Ruebel <[email protected]>

* Update docs/source/extensions_tutorial/2_create_extension_spec_walkthrough.rst

Co-authored-by: Oliver Ruebel <[email protected]>

* Update docs/source/extensions_tutorial/2_create_extension_spec_walkthrough.rst

Co-authored-by: Oliver Ruebel <[email protected]>

* Update docs/source/extensions_tutorial/3_spec_api.rst

Co-authored-by: Oliver Ruebel <[email protected]>

* Update docs/source/extensions_tutorial/3_spec_api.rst

Co-authored-by: Oliver Ruebel <[email protected]>

* Update docs/source/extensions_tutorial/6_documenting_extension.rst

Co-authored-by: Oliver Ruebel <[email protected]>

* Update docs/source/extensions_tutorial/4_auto_api.rst

Co-authored-by: Oliver Ruebel <[email protected]>

* Update docs/source/extensions_tutorial/3_spec_api.rst

Co-authored-by: Oliver Ruebel <[email protected]>

* Update docs/source/extensions_tutorial/3_spec_api.rst

Co-authored-by: Oliver Ruebel <[email protected]>

* Update docs/source/extensions_tutorial/3_spec_api.rst

Co-authored-by: Oliver Ruebel <[email protected]>

* remove detailed GroupSpec API instructions from tutorial

* remove instructions for generated the directory structure, and instead focus on generating the HTML and filling in the missing pieces.

* add link to ndx catalog

* Update docs/source/extensions_tutorial/2_create_extension_spec_walkthrough.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/3_spec_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/2_create_extension_spec_walkthrough.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/2_create_extension_spec_walkthrough.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/3_spec_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/3_spec_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/3_spec_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/3_spec_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/3_spec_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/3_spec_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/3_spec_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/3_spec_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/3_spec_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/3_spec_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/4_auto_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/5_custom_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/3_spec_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/4_auto_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/4_auto_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/4_auto_api.rst

* Update docs/source/extensions_tutorial/3_spec_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/5_custom_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update docs/source/extensions_tutorial/5_custom_api.rst

Co-authored-by: Ryan Ly <[email protected]>

* Update extensions_tutorial_home.rst

* Minor text fixes

* Minor text fixes

* Minor text fixes

Co-authored-by: Oliver Ruebel <[email protected]>
Co-authored-by: Ryan Ly <[email protected]>

Set language for code block to fix readthedocs

Add RRID for PyNWB

Add RRID to README

Update changelog

make MotionCorrection test (#1374)

* attempt to make MotionCorrection test. I don't understand why it's failing

* Fix CorrectedImageStack nwbfields

* Fix tests

* Fix test

* Fix flake8

* Fix test

Co-authored-by: Ryan Ly <[email protected]>

Add unique electrode id check (#1344)

Co-authored-by: Ryan Ly <[email protected]>

Fix integration tests with invalid test data that will be caught by future hdmf validator version (#1366)

Co-authored-by: Ryan Ly <[email protected]>

Improve documentation and test for CorrectedImageStack (#1306)

Fix test for validator update (#1376)

Update changelog

Allow h5py 3 so that this can work with hdmf rc/3.0.0

Move SweepTable DeprectationWarning to the constructor of SweepTable to clean up code

Suppress SweepTable Deprecation warning in icephys integration test

Ignore and assert deprecation warning for SweepTable during SweepTable roundtrip tests

Merge branch 'rc/2.0.0' into add/icephys_meta

Fix falke8 on icephys tutorial galleries
rly pushed a commit that referenced this pull request Aug 10, 2021
Use  extension in docs to simplify linking to common targets

Move advanced data I/O tutorials to their own section and move parallel I/O tutorial

Update Changelog for tutorials

Updated deprecation warning in SweepTable tutorial

Fix minor spelling error in make_test_files

Several corrections and enhancments for the icephys tutorial

Moved functions to create icephys test file from tests to pynwb

Moved functions to create icephys test file from tests to pynwb

Start for new tutorial to show conversion of icephys tables to pandas dataframes

Test to_hierarchical_dataframe and to_denormalized_dataframe functions

Updated tutorial to convert icephys tabels to pandas

Fix minor spelling error in NWBFile docstring

Add TimeseriesReferenceVectorData.get method to mask missing values on load

Update IntracellularRecordingsTable. Update add_recording defaults for startindex and count to None and update to_dataframe method to handle missing values in TimeSeriesReferenceVectorData

Update create_icephys_testfile to create testdata with missing stimuli

Update existing tests to match the new behavior

Update icephys tutorial for match new behavior

Update icephys pandas tutorial to demonstrate new behavior

Remove debug print statements

Add table augmentation and queries to the icephys_pandas tutorial

Fix flake8

Update to use keyword args

Clarify text in icephys tutorial

Fix flake8 on tutorial

Fix flake8

Update to use dev branch of nwb-schema

Set default name and default description for TimeSeriesReferenceVectorData

added icephys_testutils to pynwb.testing.__init__

Added test for IntracellularRecordingsTable.to_dataframe with options

Always use MaskedArrya in TimeSeriesReferenceVectorData instead of MaskedConstant

Add tests for TimeSeriesReferenceVectorData.get

Updated change log

Fix minor docstring issue in TimeSeriesReferenceData.get

Clarify docstring for TimeSeriesReferenceData.get

Fix build warnings in docs (#1380)

* Remove unused and outdated convert.rst from the docs  Fix #1378
* Fix extra numbered footnote reference in ophys tutorial
* Change code block highlighting from c to bash to avoid build warning
* Fix duplicate target here warning in 3_spec_api.rst
* Fix missing section label for crossreferencing between the extension tutorial and gallery
* Updated Changelog

Update CHANGELOG.md

Update icephys.py

Update docs/gallery/domain/plot_icephys.py

Update plot_icephys.py

Minor text fixes

Update plot_icephys_pandas.py

Minor text edits

Update icephys_testutils.py

Minor docstring edits

Update file.py

Fix typos

Update docs/gallery/domain/plot_icephys.py Replace master with main

Mention use of create_icephys_testfile function

Co-authored-by: Ryan Ly <[email protected]>

Remove old comment in TimeSeriesReferenceVectorData

Co-authored-by: Ryan Ly <[email protected]>

Fix typo in changelog

Fix broken link target

Fix flake8 in docs/gallery

Minor text fixes

Minor text and formatting fixes

Update tests/unit/test_icephys_metadata_tables.py

Co-authored-by: Ryan Ly <[email protected]>

Use namedtuple instead of numpy masked structed array to represent values of TimeSeriesReferenceVectorData

Update tests/unit/test_icephys_metadata_tables.py

Co-authored-by: Ryan Ly <[email protected]>

Update tests/unit/test_icephys_metadata_tables.py

Co-authored-by: Ryan Ly <[email protected]>

Update tests/unit/test_icephys_metadata_tables.py

Co-authored-by: Ryan Ly <[email protected]>

Update tests/unit/test_icephys_metadata_tables.py

Co-authored-by: Ryan Ly <[email protected]>

Update tests/unit/test_icephys_metadata_tables.py

Co-authored-by: Ryan Ly <[email protected]>

Update tests/unit/test_icephys_metadata_tables.py

Co-authored-by: Ryan Ly <[email protected]>

Update tests/unit/test_icephys_metadata_tables.py

Co-authored-by: Ryan Ly <[email protected]>

Update tests/unit/test_icephys_metadata_tables.py

Co-authored-by: Ryan Ly <[email protected]>

Fix bad indent in test

Update base.py - minor text edits

Use hdmf 3.1.1

Enhance introspection, slicing, and data addition for TimeSeriesReference and TimeSeriesReferenceVectorData

Updated icephys query tutorial to use latest get_linked_tables behavior and improve rendering

Fix flake8 in gallery

Fix spelling in error message

Update src/pynwb/base.py

Co-authored-by: Ryan Ly <[email protected]>

Update src/pynwb/base.py

Co-authored-by: Ryan Ly <[email protected]>

Update src/pynwb/base.py

Co-authored-by: Ryan Ly <[email protected]>

Update src/pynwb/base.py

Co-authored-by: Ryan Ly <[email protected]>

Update src/pynwb/base.py

Co-authored-by: Ryan Ly <[email protected]>

Update src/pynwb/base.py

Co-authored-by: Ryan Ly <[email protected]>

Fix bug in TimeSeriesReference.timestamps and add comments

Fix gallery tests to handle allensdk pinning pynwb/hdmf

Fix rebase
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants