Add/icephys meta

CHANGELOG.md

@@ -1,27 +1,47 @@
 # PyNWB Changelog
 
-## PyNWB 2.0.0 (TBD, 2021)
+## PyNWB 2.0.0 (Upcoming)
+
+### Breaking changes:
+- ``SweepTable`` has been deprecated in favor of the new icephys metadata tables. Use of ``SweepTable``
+  is still possible but no longer recommended. @oruebel  (#1349)
 
 ### New features:
--
--
--
-- Drop Python 3.6 support, add Python 3.9 support. @rly (#1377)
-- Update requirements to allow compatibility with HDMF 3 and h5py 3. @rly (#1377)
+- Added new intracellular electrophysiology hierarchical table structure from ndx-icephys-meta to NWB core.
+  This includes the new types ``TimeSeriesReferenceVectorData``, ``IntracellularRecordingsTable``,
+  ``SimultaneousRecordingsTable``, ``SequentialRecordingsTable``, ``RepetitionsTable`` and
+  ``ExperimentalConditionsTable`` as well as corresponding updates to ``NWBFile`` to support interaction
+   with the new tables. @oruebel  (#1349)
+- Added support for nwb-schema 2.4.0. See [Release Notes](https://nwb-schema.readthedocs.io/en/latest/format_release_notes.html)
+  for more details. @oruebel (#1349)
+- Dropped Python 3.6 support, added Python 3.9 support. @rly (#1377)
+- Updated requirements to allow compatibility with HDMF 3 and h5py 3. @rly (#1377)
+
+### Tutorial enhancements:
+- Added new tutorial for intracellular electrophysiology to describe the use of the new metadata tables
+  and declared the previous tutoral using ``SweepTable`` as deprecated.  @oruebel (#1349)
+- Added new tutorial for querying intracellular electrophysiology metadata
+  (``docs/gallery/domain/plot_icephys_pandas.py``). @oruebel (#1349, #1383)
+- Added thumbnails for tutorials to improve presentation of online docs.  @oruebel (#1349)
+- Used `sphinx.ext.extlinks` extension in docs to simplify linking to common targets. @oruebel (#1349)
+- Created new section for advanced I/O tutorials and moved parallel I/O tutorial to its own file. @oruebel (#1349)
 
 ### Minor new features:
 - Add RRID for citing PyNWB to the docs. @oruebel (#1372)
 - Update CI and tests to handle deprecations in libraries. @rly (#1377)
+- Add test utilities for icephys (``pynwb.testing.icephys_testutils``) to ease creation of test data
+  for tests and tutorials. @oruebel (#1349, #1383)
 
 ### Bug fixes:
-- Enforce electrode ID uniqueness during insertion into table. @CodyCBakerPhD (#1344)
-- Fix integration tests with invalid test data that will be caught by future hdmf validator version.
+- Updated behavior of ``make clean`` command for docs to ensure tutorial files are cleaned up.  @oruebel (#1349)
+- Enforced electrode ID uniqueness during insertion into table. @CodyCBakerPhD (#1344)
+- Fixed integration tests with invalid test data that will be caught by future hdmf validator version.
   @dsleiter, @rly (#1366, #1376)
-- Fix build warnings in docs @oruebel (#1380)
+- Fixed build warnings in docs @oruebel (#1380)
 
 ## PyNWB 1.5.1 (May 24, 2021)
 
-### Bug fix:
+### Bug fixes:
 - Raise minimum version of pandas from 0.23 to 1.0.5 to be compatible with numpy 1.20, and raise minimum version of
   HDMF to use the corresponding change in HDMF. @rly (#1363)
 - Update documentation and update structure of requirements files. @rly (#1363)
@@ -64,7 +84,7 @@
   - Add capability to add a row to a column after IO.
   - Add method `AbstractContainer.get_fields_conf`.
   - Add functionality for storing external resource references.
-  - Add method `hdmf.utils.get_docval_macro` to get a tuple of the current values for a docval_macro, e.g., 'array_data'  
+  - Add method `hdmf.utils.get_docval_macro` to get a tuple of the current values for a docval_macro, e.g., 'array_data'
     and 'scalar_data'.
   - `DynamicTable` can be automatically generated using `get_class`. Now the HDMF API can read files with extensions
     that contain a DynamicTable without needing to import the extension first.

docs/Makefile

@@ -45,7 +45,7 @@ help:
 	@echo "  apidoc     to build RST from source code"
 
 clean:
-	-rm -rf $(BUILDDIR)/* $(RSTDIR)/$(PKGNAME)*.rst
+	-rm -rf $(BUILDDIR)/* $(RSTDIR)/$(PKGNAME)*.rst $(RSTDIR)/tutorials
 
 html:
 	$(SPHINXBUILD) -b html $(ALLSPHINXOPTS) $(BUILDDIR)/html

docs/gallery/advanced_io/README.txt

@@ -0,0 +1,7 @@
+
+
+.. _general-tutorials:
+
+
+Advanced I/O
+------------

docs/gallery/general/advanced_hdf5_io.py → docs/gallery/advanced_io/h5dataio.py

@@ -1,6 +1,6 @@
 '''
-Advanced HDF5 I/O
-=====================
+Defining HDF5 Dataset I/O Settings (chunking, compression, etc.)
+================================================================
 
 The HDF5 storage backend supports a broad range of advanced dataset I/O options, such as,
 chunking and compression. Here we demonstrate how to use these features
@@ -18,6 +18,7 @@
 # Before we get started, lets create an NWBFile for testing so that we can add our data to it.
 #
 
+# sphinx_gallery_thumbnail_path = 'figures/gallery_thumbnails_h5dataio.png'
 from datetime import datetime
 from dateutil.tz import tzlocal
 from pynwb import NWBFile
@@ -216,72 +217,6 @@
 #   will be ignored as the h5py.Dataset will either be linked to or copied as on write.
 #
 
-####################
-# Parallel I/O using MPI
-# ----------------------
-#
-# The HDF5 storage backend supports parallel I/O using the Message Passing Interface (MPI).
-# Using this feature requires that you install ``hdf5`` and ``h5py`` against an MPI driver, and you
-# install ``mpi4py``. The basic installation of pynwb will not work. Setup can be tricky, and
-# is outside the scope of this tutorial (for now), and the following assumes that you have
-# HDF5 installed in a MPI configuration. Here we:
-#
-# 1. **Instantiate a dataset for parallel write**: We create TimeSeries with 4 timestamps that we
-# will write in parallel
-#
-# 2. **Write to that file in parallel using MPI**: Here we assume 4 MPI ranks while each rank writes
-# the data for a different timestamp.
-#
-# 3. **Read from the file in parallel using MPI**: Here each of the 4 MPI ranks reads one time
-# step from the file
-#
-# .. code-block:: python
-#
-#   from mpi4py import MPI
-#   import numpy as np
-#   from dateutil import tz
-#   from pynwb import NWBHDF5IO, NWBFile, TimeSeries
-#   from datetime import datetime
-#   from hdmf.data_utils import DataChunkIterator
-#
-#   start_time = datetime(2018, 4, 25, 2, 30, 3, tzinfo=tz.gettz('US/Pacific'))
-#   fname = 'test_parallel_pynwb.nwb'
-#   rank = MPI.COMM_WORLD.rank  # The process ID (integer 0-3 for 4-process run)
-#
-#   # Create file on one rank. Here we only instantiate the dataset we want to
-#   # write in parallel but we do not write any data
-#   if rank == 0:
-#       nwbfile = NWBFile('aa', 'aa', start_time)
-#       data = DataChunkIterator(data=None, maxshape=(4,), dtype=np.dtype('int'))
-#
-#       nwbfile.add_acquisition(TimeSeries('ts_name', description='desc', data=data,
-#                                          rate=100., unit='m'))
-#       with NWBHDF5IO(fname, 'w') as io:
-#           io.write(nwbfile)
-#
-#   # write to dataset in parallel
-#   with NWBHDF5IO(fname, 'a', comm=MPI.COMM_WORLD) as io:
-#       nwbfile = io.read()
-#       print(rank)
-#       nwbfile.acquisition['ts_name'].data[rank] = rank
-#
-#   # read from dataset in parallel
-#   with NWBHDF5IO(fname, 'r', comm=MPI.COMM_WORLD) as io:
-#       print(io.read().acquisition['ts_name'].data[rank])
-
-####################
-# To specify details about chunking, compression and other HDF5-specific I/O options,
-# we can wrap data via ``H5DataIO``, e.g,
-#
-# .. code-block:: python
-#
-#   data = H5DataIO(DataChunkIterator(data=None, maxshape=(100000, 100),
-#                                     dtype=np.dtype('float')),
-#                                     chunks=(10, 10), maxshape=(None, None))
-#
-# would initialize your dataset with a shape of (100000, 100) and maxshape of (None, None)
-# and your own custom chunking of (10, 10).
-
 ####################
 # Disclaimer
 # ----------------

docs/gallery/general/iterative_write.py → docs/gallery/advanced_io/iterative_write.py

@@ -100,6 +100,7 @@
 # simple helper function first to write a simple NWBFile containing a single timeseries to
 # avoid repetition of the same code and to allow us to focus on the important parts of this tutorial.
 
+# sphinx_gallery_thumbnail_path = 'figures/gallery_thumbnails_iterative_write.png'
 from datetime import datetime
 from dateutil.tz import tzlocal
 from pynwb import NWBFile, TimeSeries

docs/gallery/general/linking_data.py → docs/gallery/advanced_io/linking_data.py

@@ -56,7 +56,7 @@
 # In the following we are creating two :py:meth:`~pynwb.base.TimeSeries` each written to a separate file.
 # We then show how we can integrate these files into a single NWBFile.
 
-
+# sphinx_gallery_thumbnail_path = 'figures/gallery_thumbnails_linking_data.png'
 from datetime import datetime
 from dateutil.tz import tzlocal
 from pynwb import NWBFile

docs/gallery/advanced_io/parallelio.py

@@ -0,0 +1,81 @@
+'''
+Parallel I/O using MPI
+======================
+
+The HDF5 storage backend supports parallel I/O using the Message Passing Interface (MPI).
+Using this feature requires that you install ``hdf5`` and ``h5py`` against an MPI driver, and you
+install ``mpi4py``. The basic installation of pynwb will not work. Setup can be tricky, and
+is outside the scope of this tutorial (for now), and the following assumes that you have
+HDF5 installed in a MPI configuration.
+'''
+
+# sphinx_gallery_thumbnail_path = 'figures/gallery_thumbnails_parallelio.png'
+
+####################
+# Here we:
+#
+# 1. **Instantiate a dataset for parallel write**: We create TimeSeries with 4 timestamps that we
+# will write in parallel
+#
+# 2. **Write to that file in parallel using MPI**: Here we assume 4 MPI ranks while each rank writes
+# the data for a different timestamp.
+#
+# 3. **Read from the file in parallel using MPI**: Here each of the 4 MPI ranks reads one time
+# step from the file
+#
+# .. code-block:: python
+#
+#   from mpi4py import MPI
+#   import numpy as np
+#   from dateutil import tz
+#   from pynwb import NWBHDF5IO, NWBFile, TimeSeries
+#   from datetime import datetime
+#   from hdmf.data_utils import DataChunkIterator
+#
+#   start_time = datetime(2018, 4, 25, 2, 30, 3, tzinfo=tz.gettz('US/Pacific'))
+#   fname = 'test_parallel_pynwb.nwb'
+#   rank = MPI.COMM_WORLD.rank  # The process ID (integer 0-3 for 4-process run)
+#
+#   # Create file on one rank. Here we only instantiate the dataset we want to
+#   # write in parallel but we do not write any data
+#   if rank == 0:
+#       nwbfile = NWBFile('aa', 'aa', start_time)
+#       data = DataChunkIterator(data=None, maxshape=(4,), dtype=np.dtype('int'))
+#
+#       nwbfile.add_acquisition(TimeSeries('ts_name', description='desc', data=data,
+#                                          rate=100., unit='m'))
+#       with NWBHDF5IO(fname, 'w') as io:
+#           io.write(nwbfile)
+#
+#   # write to dataset in parallel
+#   with NWBHDF5IO(fname, 'a', comm=MPI.COMM_WORLD) as io:
+#       nwbfile = io.read()
+#       print(rank)
+#       nwbfile.acquisition['ts_name'].data[rank] = rank
+#
+#   # read from dataset in parallel
+#   with NWBHDF5IO(fname, 'r', comm=MPI.COMM_WORLD) as io:
+#       print(io.read().acquisition['ts_name'].data[rank])
+
+####################
+# To specify details about chunking, compression and other HDF5-specific I/O options,
+# we can wrap data via ``H5DataIO``, e.g,
+#
+# .. code-block:: python
+#
+#   data = H5DataIO(DataChunkIterator(data=None, maxshape=(100000, 100),
+#                                     dtype=np.dtype('float')),
+#                                     chunks=(10, 10), maxshape=(None, None))
+#
+# would initialize your dataset with a shape of (100000, 100) and maxshape of (None, None)
+# and your own custom chunking of (10, 10).
+
+####################
+# Disclaimer
+# ----------------
+#
+# External links included in the tutorial are being provided as a convenience and for informational purposes only;
+# they do not constitute an endorsement or an approval by the authors of any of the products, services or opinions of
+# the corporation or organization or individual. The authors bear no responsibility for the accuracy, legality or
+# content of the external site or for that of subsequent links. Contact the external site for answers to questions
+# regarding its content.

docs/gallery/domain/brain_observatory.py

@@ -10,10 +10,13 @@
 # physiology submodule (pynwb.ophys). We will use the allensdk as a read API, while leveraging the pynwb data model and
 # write api to transform and write the data back to disk.
 #
+# .. note: Using the latest allensdk package requires Python 3.6 or higher.
+
 ########################################
 # .. raw:: html
 #     :url: https://gist.githubusercontent.com/nicain/82e6b3d8f9ff5b85ef01a582e41e2389/raw/
 
+# sphinx_gallery_thumbnail_path = 'figures/gallery_thumbnails_allenbrainobservatory.png'
 from datetime import datetime
 from dateutil.tz import tzlocal
 

docs/gallery/domain/ecephys.py

@@ -5,20 +5,18 @@
 Extracellular electrophysiology data
 ============================================
 
-The following examples will reference variables that may not be defined within the block they are used in. For
-clarity, we define them here:
 '''
 
-
-import numpy as np
-
 #######################
 # Creating and Writing NWB files
 # ------------------------------
 #
 # When creating a NWB file, the first step is to create the :py:class:`~pynwb.file.NWBFile`. The first
 # argument is the name of the NWB file, and the second argument is a brief description of the dataset.
 
+# sphinx_gallery_thumbnail_path = 'figures/gallery_thumbnails_ecephys.png'
+
+import numpy as np
 from datetime import datetime
 from dateutil.tz import tzlocal
 from pynwb import NWBFile

docs/gallery/domain/icephys.py

@@ -2,11 +2,17 @@
 '''
 .. _icephys_tutorial:
 
-Intracellular electrophysiology data
-============================================
+Intracellular electrophysiology data using SweepTable
+=====================================================
 
-The following examples will reference variables that may not be defined within the block they are used in. For
-clarity, we define them here:
+The following tutorial describes storage of intracellular electrophysiology data in NWB using the
+SweepTable to manage recordings.
+
+.. warning::
+    The use of SweepTable has been deprecated as of PyNWB >v2.0 in favor of the new hierarchical
+    intracellular electrophysiology metadata tables to allow for a more complete description of
+    intracellular electrophysiology experiments. See the :doc:`Intracellular electrophysiology  <plot_icephys>`
+    tutorial for details.
 '''
 
 #######################
@@ -16,6 +22,7 @@
 # When creating a NWB file, the first step is to create the :py:class:`~pynwb.file.NWBFile`. The first
 # argument is is a brief description of the dataset.
 
+# sphinx_gallery_thumbnail_path = 'figures/gallery_thumbnails_icephys_sweeptable.png'
 from datetime import datetime
 from dateutil.tz import tzlocal
 from pynwb import NWBFile
@@ -76,7 +83,7 @@
 ccss = CurrentClampStimulusSeries(
     name="ccss", data=[1, 2, 3, 4, 5], starting_time=123.6, rate=10e3, electrode=elec, gain=0.02, sweep_number=0)
 
-nwbfile.add_stimulus(ccss)
+nwbfile.add_stimulus(ccss, use_sweep_table=True)
 
 #######################
 # We now add another stimulus series but from a different sweep. TimeSeries
@@ -87,7 +94,7 @@
 vcss = VoltageClampStimulusSeries(
     name="vcss", data=[2, 3, 4, 5, 6], starting_time=234.5, rate=10e3, electrode=elec, gain=0.03, sweep_number=1)
 
-nwbfile.add_stimulus(vcss)
+nwbfile.add_stimulus(vcss, use_sweep_table=True)
 
 #######################
 # Here, we will use :py:class:`~pynwb.icephys.CurrentClampSeries` to store current clamp
@@ -102,7 +109,7 @@
     electrode=elec, gain=0.02, bias_current=1e-12, bridge_balance=70e6,
     capacitance_compensation=1e-12, sweep_number=0)
 
-nwbfile.add_acquisition(ccs)
+nwbfile.add_acquisition(ccs, use_sweep_table=True)
 
 #######################
 # And voltage clamp data from the second sweep using
@@ -116,7 +123,7 @@
     electrode=elec, gain=0.02, capacitance_slow=100e-12, resistance_comp_correction=70.0,
     sweep_number=1)
 
-nwbfile.add_acquisition(vcs)
+nwbfile.add_acquisition(vcs, use_sweep_table=True)
 
 ####################
 # .. _icephys_writing:
@@ -183,7 +190,7 @@
 # PatchClampSeries which belongs to a certain sweep number via
 # :py:meth:`~pynwb.icephys.SweepTable.get_series`.
 #
-# The following call will return the voltage clamp data, two timeseries
+# The following call will return the voltage clamp data of two timeseries
 # consisting of acquisition and stimulus, from sweep 1.
 
 series = nwbfile.sweep_table.get_series(1)

docs/gallery/domain/ophys.py

@@ -17,6 +17,7 @@
 clarity, we define them here:
 '''
 
+# sphinx_gallery_thumbnail_path = 'figures/gallery_thumbnails_ophys.png'
 from datetime import datetime
 from dateutil.tz import tzlocal