Merge branch 'dev' into doc/df_over_f

.github/ISSUE_TEMPLATE/bug_report.md

@@ -22,7 +22,8 @@ assignees: ''
     Python Executable: Conda or Python
     Python Version: Python 3.5, 3.6, or 3.7
     Operating System: Windows, macOS or Linux
-    HDMF Version: Version of PyNWB used
+    HDMF Version: 
+    PyNWB Version:
 
 ## Checklist
 

CHANGELOG.md

@@ -1,19 +1,14 @@
 # PyNWB Changelog
 
-## PyNWB 2.0.0 (Upcoming)
-
-### Breaking changes:
-- The 'notes' and 'table_description' arguments of `NWBFile.add_scratch(...)` are now replaced by a 'description'
-  argument that is required when a scalar, numpy.ndarray, list, tuple, or pandas.DataFrame is added to scratch. The
-  'notes' argument of `ScratchData.__init__(...)` is now replaced by the required 'description' argument for
-  consistency. Previously, 'notes' had a default value of empty string, which is not recommended. @rly (#1309)
+## PyNWB 1.5.0 (April 23, 2021)
 
 ### New features:
 - `NWBFile.add_scratch(...)` and `ScratchData.__init__(...)` now accept scalar data in addition to the currently
   accepted types. @rly (#1309)
 - Support `pathlib.Path` paths when opening files with `NWBHDF5IO`. @dsleiter (#1314)
-- Use HDMF 2.4.0. See the [HDMF 2.4.0 release notes](https://github.com/hdmf-dev/hdmf/releases/tag/2.4.0) for details.
+- Use HDMF 2.5.1. See the [HDMF release notes](https://github.com/hdmf-dev/hdmf/releases/tag/2.5.1) for details.
 - Support `driver='ros3'` in `NWBHDF5IO` for streaming NWB files directly from s3. @bendichter (#1331)
+........ TODO
 
 ## PyNWB 1.4.0 (August 12, 2020)
 

Legal.txt

@@ -1,4 +1,4 @@
-“pynwb” Copyright (c) 2017-2020, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).  All rights reserved.
+“pynwb” Copyright (c) 2017-2021, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).  All rights reserved.
 
 If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Innovation & Partnerships Office at [email protected].
 

README.rst

@@ -43,6 +43,14 @@ Overall Health
 .. image:: https://requires.io/github/NeurodataWithoutBorders/pynwb/requirements.svg?branch=dev
      :target: https://requires.io/github/NeurodataWithoutBorders/pynwb/requirements/?branch=dev
      :alt: Requirements Status
+
+.. image:: https://readthedocs.org/projects/pynwb/badge/?version=latest
+     :target: https://pynwb.readthedocs.io/en/latest/?badge=latest
+     :alt: Documentation Status
+
+.. image:: https://img.shields.io/pypi/l/pynwb.svg
+     :target: https://github.com/neurodatawithoutborders/pynwb/blob/dev/license.txt
+     :alt:    PyPI - License
 
 NWB Format API
 ==============
@@ -77,7 +85,7 @@ For details on how to contribute to PyNWB see our `contribution guidelines <docs
 LICENSE
 =======
 
-"pynwb" Copyright (c) 2017-2019, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).  All rights reserved.
+"pynwb" Copyright (c) 2017-2021, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).  All rights reserved.
 Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
 
 (1) Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
@@ -93,7 +101,7 @@ You are under no obligation whatsoever to provide any bug fixes, patches, or upg
 COPYRIGHT
 =========
 
-"pynwb" Copyright (c) 2017-2020, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).  All rights reserved.
+"pynwb" Copyright (c) 2017-2021, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).  All rights reserved.
 If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Innovation & Partnerships Office at [email protected].
 
 NOTICE.  This Software was developed under funding from the U.S. Department of Energy and the U.S. Government consequently retains certain rights. As such, the U.S. Government has been granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the Software to reproduce, distribute copies to the public, prepare derivative works, and perform publicly and display publicly, and to permit other to do so.

docs/source/conf.py

@@ -93,7 +93,7 @@
 
 # General information about the project.
 project = u'PyNWB'
-copyright = u'2017-2020, Neurodata Without Borders'
+copyright = u'2017-2021, Neurodata Without Borders'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the

license.txt

@@ -1,4 +1,4 @@
-“pynwb” Copyright (c) 2017-2020, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).  All rights reserved.
+“pynwb” Copyright (c) 2017-2021, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).  All rights reserved.
 
 Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
 

requirements-min.txt

@@ -1,6 +1,8 @@
-# these minimum requirements specify '==' for testing; setup.py replaces '==' with '>='
-h5py==2.9  # support for setting attrs to lists of utf-8 added in 2.9
-hdmf==2.4
-numpy==1.16
-pandas==0.23
-python-dateutil==2.7
+# package dependencies and their minimum versions for installing PyNWB
+# the requirements here specify '==' for testing; setup.py replaces '==' with '>='
+h5py==2.9,<3  # support for setting attrs to lists of utf-8 added in 2.9
+hdmf==2.5.2,<3
+numpy==1.16,<1.21
+pandas==0.23,<2
+python-dateutil==2.7,<3
+setuptools

requirements.txt

@@ -1,5 +1,6 @@
 h5py==2.10.0
-hdmf==2.4.0
+hdmf==2.5.2
 numpy==1.18.5
 pandas==0.25.3
 python-dateutil==2.8.1
+setuptools==56.0.0

src/pynwb/__init__.py

@@ -41,10 +41,8 @@ def _get_resources():
 __NS_CATALOG = NamespaceCatalog(NWBGroupSpec, NWBDatasetSpec, NWBNamespace)
 
 hdmf_typemap = hdmf.common.get_type_map()
-__NS_CATALOG.merge(hdmf_typemap.namespace_catalog)
-
 __TYPE_MAP = TypeMap(__NS_CATALOG)
-__TYPE_MAP.merge(hdmf_typemap)
+__TYPE_MAP.merge(hdmf_typemap, ns_catalog=True)
 
 
 @docval({'name': 'extensions', 'type': (str, TypeMap, list),
@@ -185,7 +183,7 @@ def get_sum(self, a, b):
 
     """
     neurodata_type, namespace = getargs('neurodata_type', 'namespace', kwargs)
-    return __TYPE_MAP.get_container_cls(namespace, neurodata_type)
+    return __TYPE_MAP.get_dt_container_cls(neurodata_type, namespace)
 
 
 @docval({'name': 'io', 'type': HDMFIO, 'doc': 'the HDMFIO object to read from'},

src/pynwb/core.py

@@ -1,4 +1,5 @@
 import numpy as np
+from warnings import warn
 
 from hdmf import Container, Data
 from hdmf.container import AbstractContainer, MultiContainerInterface as hdmf_MultiContainerInterface, Table
@@ -88,14 +89,39 @@ def extend(self, arg):
 @register_class('ScratchData', CORE_NAMESPACE)
 class ScratchData(NWBData):
 
-    __nwbfields__ = ('description',)
+    __nwbfields__ = ('description', )
 
     @docval({'name': 'name', 'type': str, 'doc': 'the name of this container'},
             {'name': 'data', 'type': ('scalar_data', 'array_data', 'data', Data), 'doc': 'the source of the data'},
-            {'name': 'description', 'type': str, 'doc': 'description of the data'})
+            {'name': 'notes', 'type': str,
+             'doc': 'notes about the data. This argument will be deprecated. Use description instead', 'default': ''},
+            {'name': 'description', 'type': str, 'doc': 'notes about the data', 'default': None})
     def __init__(self, **kwargs):
         call_docval_func(super().__init__, kwargs)
-        self.description = getargs('description', kwargs)
+        notes, description = getargs('notes', 'description', kwargs)
+        if notes != '':
+            warn('The `notes` argument of ScratchData.__init__ will be deprecated. Use description instead.',
+                 PendingDeprecationWarning)
+            if notes != '' and description != '':
+                raise ValueError('Cannot provide both notes and description to ScratchData.__init__. The description '
+                                 'argument is recommended.')
+            description = notes
+        if not description:
+            warn('ScratchData.description will be required in a future major release of PyNWB.',
+                 PendingDeprecationWarning)
+        self.description = description
+
+    @property
+    def notes(self):
+        warn('Use of ScratchData.notes will be deprecated. Use ScratchData.description instead.',
+             PendingDeprecationWarning)
+        return self.description
+
+    @notes.setter
+    def notes(self, value):
+        warn('Use of ScratchData.notes will be deprecated. Use ScratchData.description instead.',
+             PendingDeprecationWarning)
+        self.description = value
 
 
 class NWBTable(Table):

src/pynwb/file.py

@@ -674,29 +674,51 @@ def add_stimulus_template(self, timeseries):
              'type': ('scalar_data', np.ndarray, list, tuple, pd.DataFrame, DynamicTable, NWBContainer, ScratchData),
              'doc': 'The data to add to the scratch space.'},
             {'name': 'name', 'type': str,
-             'doc': ('The name of the data. Required only when passing in a scalar, numpy.ndarray, '
-                     'list, tuple, or pandas.DataFrame'),
+             'doc': 'The name of the data. Required only when passing in a scalar, numpy.ndarray, list, or tuple',
              'default': None},
+            {'name': 'notes', 'type': str,
+             'doc': ('Notes to add to the data. Only used when passing in numpy.ndarray, list, or tuple. This '
+                     'argument is not recommended. Use the `description` argument instead.'),
+             'default': None},
+            {'name': 'table_description', 'type': str,
+             'doc': ('Description for the internal DynamicTable used to store a pandas.DataFrame. This '
+                     'argument is not recommended. Use the `description` argument instead.'),
+             'default': ''},
             {'name': 'description', 'type': str,
              'doc': ('Description of the data. Required only when passing in a scalar, numpy.ndarray, '
                      'list, tuple, or pandas.DataFrame. Ignored when passing in an NWBContainer, '
                      'DynamicTable, or ScratchData object.'),
              'default': None})
     def add_scratch(self, **kwargs):
-        '''Add data to the scratch space.'''
-        data, name, description = getargs('data', 'name', 'description', kwargs)
+        '''Add data to the scratch space'''
+        data, name, notes, table_description, description = getargs('data', 'name', 'notes', 'table_description',
+                                                                    'description', kwargs)
+        if notes is not None or table_description != '':
+            warn('Use of the `notes` or `table_description` argument will be removed in a future version of PyNWB. '
+                 'Use the `description` argument instead.', PendingDeprecationWarning)
+            if description is not None:
+                raise ValueError('Cannot call add_scratch with (notes or table_description) and description')
+
         if isinstance(data, (str, int, float, bytes, np.ndarray, list, tuple, pd.DataFrame)):
             if name is None:
                 msg = ('A name is required for NWBFile.add_scratch when adding a scalar, numpy.ndarray, '
                        'list, tuple, or pandas.DataFrame as scratch data.')
                 raise ValueError(msg)
-            if description is None:
-                msg = ('A description is required for NWBFile.add_scratch when adding a scalar, numpy.ndarray, '
-                       'list, tuple, or pandas.DataFrame as scratch data.')
-                raise ValueError(msg)
             if isinstance(data, pd.DataFrame):
+                if table_description != '':
+                    description = table_description  # remove after deprecation
+                if description is None:
+                    msg = ('A description is required for NWBFile.add_scratch when adding a scalar, numpy.ndarray, '
+                           'list, tuple, or pandas.DataFrame as scratch data.')
+                    raise ValueError(msg)
                 data = DynamicTable.from_dataframe(df=data, name=name, table_description=description)
             else:
+                if notes is not None:
+                    description = notes  # remove after deprecation
+                if description is None:
+                    msg = ('A description is required for NWBFile.add_scratch when adding a scalar, numpy.ndarray, '
+                           'list, tuple, or pandas.DataFrame as scratch data.')
+                    raise ValueError(msg)
                 data = ScratchData(name=name, data=data, description=description)
         else:
             if name is not None:
@@ -707,8 +729,8 @@ def add_scratch(self, **kwargs):
                      'DynamicTable to scratch.')
         return self._add_scratch(data)
 
-    @docval({'name': 'name', 'type': str, 'help': 'the name of the object to get'},
-            {'name': 'convert', 'type': bool, 'help': 'return the original data, not the NWB object', 'default': True})
+    @docval({'name': 'name', 'type': str, 'doc': 'the name of the object to get'},
+            {'name': 'convert', 'type': bool, 'doc': 'return the original data, not the NWB object', 'default': True})
     def get_scratch(self, **kwargs):
         '''Get data from the scratch space'''
         name, convert = getargs('name', 'convert', kwargs)

tests/integration/test_io.py

@@ -275,8 +275,7 @@ def test_append(self):
             self.assertIs(nwb.processing['test_proc_mod']['LFP'].electrical_series['test_es'].electrodes,
                           nwb.acquisition['timeseries2'].electrodes)
             errors = validate(io)
-            for e in errors:
-                print('ERROR', e)
+            self.assertTrue(len(errors) == 0)
 
 
 class TestH5DataIO(TestCase):

tests/unit/test_core.py

@@ -81,4 +81,4 @@ def test_print_file(self):
 
 class TestAvailableNamespaces(TestCase):
     def test_available_namespaces(self):
-        self.assertEqual(available_namespaces(), ('hdmf-common', 'core'))
+        self.assertEqual(available_namespaces(), ('hdmf-common', 'hdmf-experimental', 'core'))

tests/unit/test_extension.py

@@ -49,7 +49,7 @@ def test_load_namespace(self):
     def test_get_class(self):
         self.test_export()
         type_map = get_type_map(extensions=os.path.join(self.tempdir, self.ns_path))
-        type_map.get_container_cls(self.prefix, 'TetrodeSeries')
+        type_map.get_dt_container_cls('TetrodeSeries', self.prefix)
 
     def test_load_namespace_with_reftype_attribute(self):
         ns_builder = NWBNamespaceBuilder('Extension for use in my Lab', self.prefix, version='0.1.0')
@@ -74,7 +74,7 @@ def test_load_namespace_with_reftype_attribute_check_autoclass_const(self):
         ns_builder.add_spec(self.ext_source, test_ds_ext)
         ns_builder.export(self.ns_path, outdir=self.tempdir)
         type_map = get_type_map(extensions=os.path.join(self.tempdir, self.ns_path))
-        my_new_type = type_map.get_container_cls(self.prefix, 'my_new_type')
+        my_new_type = type_map.get_dt_container_cls('my_new_type', self.prefix)
         docval = None
         for tmp in get_docval(my_new_type.__init__):
             if tmp['name'] == 'target_ds':
@@ -169,28 +169,3 @@ def test_catch_dup_name(self):
         ns_builder2.export(self.ns_path2, outdir=self.tempdir)
         type_map = get_type_map(extensions=os.path.join(self.tempdir, self.ns_path1))
         type_map.load_namespaces(os.path.join(self.tempdir, self.ns_path2))
-
-
-class TestCatchDuplicateSpec(TestCase):
-
-    def setUp(self):
-        self.prefix = id_generator()
-        self.ext_source = '%s_extension3.yaml' % self.prefix
-
-    def tearDown(self):
-        pass
-
-    def test_catch_duplicate_spec(self):
-        spec1 = NWBGroupSpec("This is my new group 1",
-                             "Group1",
-                             neurodata_type_inc="NWBDataInterface",
-                             neurodata_type_def="Group1")
-        spec2 = NWBGroupSpec("This is my new group 2",
-                             "Group2",
-                             groups=[spec1],
-                             neurodata_type_inc="NWBDataInterface",
-                             neurodata_type_def="Group2")
-        ns_builder = NWBNamespaceBuilder("Example namespace", "pynwb_test_ext", version='0.1.0')
-        ns_builder.add_spec(self.ext_source, spec1)
-        with self.assertRaises(ValueError):
-            ns_builder.add_spec(self.ext_source, spec2)

tox.ini

@@ -136,22 +136,28 @@ basepython = python3.9
 deps = {[testenv:gallery]deps}
 commands = {[testenv:gallery]commands}
 
+
+# Test with python 3.8, pinned dev and doc reqs, and upgraded run requirements
 [testenv:gallery-py38-upgrade-dev]
 basepython = python3.8
 install_command =
     pip install -U -e . {opts} {packages}
 deps =
     -rrequirements-dev.txt
+    -rrequirements-doc.txt
 commands = {[testenv:gallery]commands}
 
+# Test with python 3.8, pinned dev and doc reqs, and pre-release run requirements
 [testenv:gallery-py38-upgrade-dev-pre]
 basepython = python3.8
 install_command =
     pip install -U --pre -e . {opts} {packages}
 deps =
     -rrequirements-dev.txt
+    -rrequirements-doc.txt
 commands = {[testenv:gallery]commands}
 
+# Test with python 3.6, pinned dev reqs, and minimum run requirements
 [testenv:gallery-py36-min-req]
 basepython = python3.6
 deps =
@@ -160,8 +166,8 @@ deps =
     -rrequirements-doc.txt
 commands = {[testenv:gallery]commands}
 
-[testenv:validation-py37]
-basepython = python3.7
+[testenv:validation-py38]
+basepython = python3.8
 install_command =
     pip install -U {opts} {packages}
 deps =