Add support for NWB schema 2.6.0 and prepare PyNWB 2.3.0

.github/workflows/run_coverage.yml

@@ -71,7 +71,7 @@ jobs:
 
       - name: Run integration tests and generate coverage report
         run: |
-          python -m coverage run -p test.py --integration
+          python -m coverage run -p test.py --integration --backwards
           # validation CLI tests generate separate .coverage files that need to be merged
           python -m coverage combine
           python -m coverage xml  # codecov uploader requires xml format

CHANGELOG.md

@@ -1,9 +1,12 @@
 # PyNWB Changelog
 
-## Upcoming
+## PyNWB 2.3.0 (February 23, 2023)
 
 ### Enhancements and minor changes
-- `Subject.age` can be input as a `timedelta`. @bendichter [#1590](https://github.com/NeurodataWithoutBorders/pynwb/pull/1590)
+- Added support for NWB Schema 2.6.0. @mavaylon1 [#1636](https://github.com/NeurodataWithoutBorders/pynwb/pull/1636)
+- Added a class and tests for the `OnePhotonSeries` new in NWB v2.6.0. @CodyCBakerPhD [#1593](https://github.com/NeurodataWithoutBorders/pynwb/pull/1593)(see also NWB Schema [#523](https://github.com/NeurodataWithoutBorders/nwb-schema/pull/523)
+- `Subject.age` can be input as a `timedelta` type. @bendichter [#1590](https://github.com/NeurodataWithoutBorders/pynwb/pull/1590)
+- Added `Subject.age__reference` field. @bendichter ([#1540](https://github.com/NeurodataWithoutBorders/pynwb/pull/1540))
 - `IntracellularRecordingsTable.add_recording`: the `electrode` arg is now optional, and is automatically populated from the stimulus or response.
   [#1597](https://github.com/NeurodataWithoutBorders/pynwb/pull/1597)
 - Added module `pynwb.testing.mock.icephys` and corresponding tests. @bendichter
@@ -13,6 +16,7 @@
 - Added ``NWBHDF5IO.nwb_version`` property to get the NWB version from an NWB HDF5 file @oruebel [#1612](https://github.com/NeurodataWithoutBorders/pynwb/pull/1612)
 - Updated ``NWBHDF5IO.read`` to check NWB version before read and raise more informative error if an unsupported version is found @oruebel [#1612](https://github.com/NeurodataWithoutBorders/pynwb/pull/1612)
 - Added the `driver` keyword argument to the `pynwb.validate` function as well as the corresponding namespace caching. @CodyCBakerPhD [#1588](https://github.com/NeurodataWithoutBorders/pynwb/pull/1588)
+- Updated HDMF requirement to version 3.5.1. [#1611](https://github.com/NeurodataWithoutBorders/pynwb/pull/1611)
 - Increased the stacklevel of the warning from `_add_missing_timezone` in `pynwb.file` to make identification of which datetime field is missing a timezone easier. @CodyCBakerPhD [#1641](https://github.com/NeurodataWithoutBorders/pynwb/pull/1641)
 
 ### Documentation and tutorial enhancements:

Legal.txt

@@ -1,4 +1,4 @@
-“pynwb” Copyright (c) 2017-2022, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).  All rights reserved.
+“pynwb” Copyright (c) 2017-2023, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).  All rights reserved.
 
 If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Innovation & Partnerships Office at [email protected].
 

README.rst

@@ -112,7 +112,7 @@ Citing NWB
 LICENSE
 =======
 
-"pynwb" Copyright (c) 2017-2022, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).  All rights reserved.
+"pynwb" Copyright (c) 2017-2023, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).  All rights reserved.
 Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
 
 (1) Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
@@ -128,7 +128,7 @@ You are under no obligation whatsoever to provide any bug fixes, patches, or upg
 COPYRIGHT
 =========
 
-"pynwb" Copyright (c) 2017-2022, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).  All rights reserved.
+"pynwb" Copyright (c) 2017-2023, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).  All rights reserved.
 If you have questions about your rights to use or distribute this software, please contact Berkeley Lab's Innovation & Partnerships Office at [email protected].
 
 NOTICE.  This Software was developed under funding from the U.S. Department of Energy and the U.S. Government consequently retains certain rights. As such, the U.S. Government has been granted for itself and others acting on its behalf a paid-up, nonexclusive, irrevocable, worldwide license in the Software to reproduce, distribute copies to the public, prepare derivative works, and perform publicly and display publicly, and to permit other to do so.

docs/source/conf.py

@@ -146,13 +146,13 @@ def __call__(self, filename):
     'fsspec': ("https://filesystem-spec.readthedocs.io/en/latest/", None),
 }
 
-extlinks = {'incf_lesson': ('https://training.incf.org/lesson/%s', ''),
-            'incf_collection': ('https://training.incf.org/collection/%s', ''),
-            'nwb_extension': ('https://github.com/nwb-extensions/%s', ''),
-            'pynwb': ('https://github.com/NeurodataWithoutBorders/pynwb/%s', ''),
-            'nwb_overview': ('https://nwb-overview.readthedocs.io/en/latest/%s', ''),
-            'hdmf-docs': ('https://hdmf.readthedocs.io/en/stable/%s', ''),
-            'dandi': ('https://www.dandiarchive.org/%s', '')}
+extlinks = {'incf_lesson': ('https://training.incf.org/lesson/%s', '%s'),
+            'incf_collection': ('https://training.incf.org/collection/%s', '%s'),
+            'nwb_extension': ('https://github.com/nwb-extensions/%s', '%s'),
+            'pynwb': ('https://github.com/NeurodataWithoutBorders/pynwb/%s', '%s'),
+            'nwb_overview': ('https://nwb-overview.readthedocs.io/en/latest/%s', '%s'),
+            'hdmf-docs': ('https://hdmf.readthedocs.io/en/stable/%s', '%s'),
+            'dandi': ('https://www.dandiarchive.org/%s', '%s')}
 
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
@@ -169,7 +169,7 @@ def __call__(self, filename):
 
 # General information about the project.
 project = u'PyNWB'
-copyright = u'2017-2022, Neurodata Without Borders'
+copyright = u'2017-2023, Neurodata Without Borders'
 
 # The version info for the project you're documenting, acts as replacement for
 # |version| and |release|, also used in various other places throughout the

license.txt

@@ -1,4 +1,4 @@
-“pynwb” Copyright (c) 2017-2022, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).  All rights reserved.
+“pynwb” Copyright (c) 2017-2023, The Regents of the University of California, through Lawrence Berkeley National Laboratory (subject to receipt of any required approvals from the U.S. Dept. of Energy).  All rights reserved.
 
 Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
 

requirements-min.txt

@@ -1,6 +1,6 @@
 # minimum versions of package dependencies for installing PyNWB
 h5py==2.10  # support for selection of datasets with list of indices added in 2.10
-hdmf==3.4.0
+hdmf==3.5.1
 numpy==1.16
 pandas==1.1.5
 python-dateutil==2.7.3

requirements.txt

@@ -1,6 +1,6 @@
 # pinned dependencies to reproduce an entire development environment to use PyNWB
 h5py==3.7.0
-hdmf==3.4.6
+hdmf==3.5.1
 numpy==1.23.3;python_version>='3.8'
 numpy==1.21.5;python_version<'3.8'  # note that numpy 1.22 dropped python 3.7 support
 pandas==1.5.0;python_version>='3.8'

setup.py

@@ -20,7 +20,7 @@
 
 reqs = [
     'h5py>=2.10,<4',
-    'hdmf>=3.4.2,<4',
+    'hdmf>=3.5.1,<4',
     'numpy>=1.16,<1.24',
     'pandas>=1.1.5,<2',
     'python-dateutil>=2.7.3,<3',

src/pynwb/file.py

@@ -55,6 +55,7 @@ class Subject(NWBContainer):
 
     __nwbfields__ = (
         'age',
+        "age__reference",
         'description',
         'genotype',
         'sex',
@@ -73,8 +74,20 @@ class Subject(NWBContainer):
                    'A timedelta will automatically be converted to The ISO 8601 Duration format.',
             "default": None,
         },
-        {'name': 'description', 'type': str,
-         'doc': 'A description of the subject, e.g., "mouse A10".', 'default': None},
+        {
+            "name": "age__reference",
+            "type": str,
+            "doc": "Age is with reference to this event. Can be 'birth' or 'gestational'. If reference is omitted, "
+                   "then 'birth' is implied. Value can be None when read from an NWB file with schema version "
+                   "2.0 to 2.5 where age__reference is missing.",
+            "default": "birth",
+        },
+        {
+            "name": "description",
+            "type": str,
+            "doc": 'A description of the subject, e.g., "mouse A10".',
+            "default": None,
+        },
         {'name': 'genotype', 'type': str,
          'doc': 'The genotype of the subject, e.g., "Sst-IRES-Cre/wt;Ai32(RCL-ChR2(H134R)_EYFP)/wt".',
          'default': None},
@@ -95,18 +108,30 @@ class Subject(NWBContainer):
         {'name': 'strain', 'type': str, 'doc': 'The strain of the subject, e.g., "C57BL/6J"', 'default': None},
     )
     def __init__(self, **kwargs):
-        keys_to_set = ("age",
-                       "description",
-                       "genotype",
-                       "sex",
-                       "species",
-                       "subject_id",
-                       "weight",
-                       "date_of_birth",
-                       "strain")
+        keys_to_set = (
+            "age",
+            "age__reference",
+            "description",
+            "genotype",
+            "sex",
+            "species",
+            "subject_id",
+            "weight",
+            "date_of_birth",
+            "strain",
+        )
         args_to_set = popargs_to_dict(keys_to_set, kwargs)
-        kwargs['name'] = 'subject'
-        super().__init__(**kwargs)
+        super().__init__(name="subject", **kwargs)
+
+        # NOTE when the Subject I/O mapper (see pynwb.io.file.py) reads an age__reference value of None from an
+        # NWB 2.0-2.5 file, it sets the value to "unspecified" so that when Subject.__init__ is called, the incoming
+        # age__reference value is NOT replaced by the default value ("birth") specified in the docval.
+        # then we replace "unspecified" with None here. the user will never see the value "unspecified".
+        # the ONLY way that age__reference can now be None is if it is read as None from an NWB 2.0-2.5 file.
+        if self._in_construct_mode and args_to_set["age__reference"] == "unspecified":
+            args_to_set["age__reference"] = None
+        elif args_to_set["age__reference"] not in ("birth", "gestational"):
+            raise ValueError("age__reference, if supplied, must be 'birth' or 'gestational'.")
 
         weight = args_to_set['weight']
         if isinstance(weight, float):

src/pynwb/io/file.py

@@ -5,6 +5,7 @@
 from .. import register_map
 from ..file import NWBFile, Subject
 from ..core import ScratchData
+from .utils import get_nwb_version
 
 
 @register_map(NWBFile)
@@ -220,3 +221,16 @@ def dateconversion(self, builder, manager):
             datestr = dob_builder.data
             date = dateutil_parse(datestr)
             return date
+
+    @ObjectMapper.constructor_arg("age__reference")
+    def age_reference_none(self, builder, manager):
+        age_builder = builder.get("age")
+        age_reference = None
+        if age_builder is not None:
+            age_reference = age_builder["attributes"].get("reference")
+        if age_reference is None:
+            if get_nwb_version(builder) < (2, 6, 0):
+                return "unspecified"  # this is handled specially in Subject.__init__
+            else:
+                return "birth"
+        return age_reference

src/pynwb/io/utils.py

@@ -0,0 +1,36 @@
+import re
+from typing import Tuple
+
+from hdmf.build import Builder
+
+
+def get_nwb_version(builder: Builder, include_prerelease=False) -> Tuple[int, ...]:
+    """Get the version of the NWB file from the root of the given builder, as a tuple.
+
+    If the "nwb_version" attribute on the root builder equals "2.5.1", then (2, 5, 1) is returned.
+    If the "nwb_version" attribute on the root builder equals "2.5.1-alpha" and include_prerelease=False,
+    then (2, 5, 1) is returned.
+    If the "nwb_version" attribute on the root builder equals "2.5.1-alpha" and include_prerelease=True,
+    then (2, 5, 1, "alpha") is returned.
+
+    :param builder: Any builder within an NWB file.
+    :type builder: Builder
+    :param include_prerelease: Whether to include prerelease information in the returned tuple.
+    :type include_prerelease: bool
+    :return: The version of the NWB file, as a tuple.
+    :rtype: tuple
+    :raises ValueError: if the 'nwb_version' attribute is missing from the root of the NWB file.
+    """
+    temp_builder = builder
+    while temp_builder.parent is not None:
+        temp_builder = temp_builder.parent
+    root_builder = temp_builder
+    nwb_version = root_builder.attributes.get("nwb_version")
+    if nwb_version is None:
+        raise ValueError("'nwb_version' attribute is missing from the root of the NWB file.")
+    nwb_version_match = re.match(r"(\d+\.\d+\.\d+)", nwb_version)[0]  # trim off any non-numeric symbols at end
+    version_list = [int(i) for i in nwb_version_match.split(".")]
+    if include_prerelease:
+        prerelease_info = nwb_version[nwb_version.index("-")+1:]
+        version_list.append(prerelease_info)
+    return tuple(version_list)

src/pynwb/ophys.py

@@ -120,6 +120,87 @@ def __init__(self, **kwargs):
             setattr(self, key, val)
 
 
+@register_class("OnePhotonSeries", CORE_NAMESPACE)
+class OnePhotonSeries(ImageSeries):
+    """Image stack recorded over time from 1-photon microscope."""
+
+    __nwbfields__ = (
+        "imaging_plane", "pmt_gain", "scan_line_rate", "exposure_time", "binning", "power", "intensity"
+    )
+
+    @docval(
+        *get_docval(ImageSeries.__init__, "name"),  # required
+        {"name": "imaging_plane", "type": ImagingPlane, "doc": "Imaging plane class/pointer."},  # required
+        *get_docval(ImageSeries.__init__, "data", "unit", "format"),
+        {"name": "pmt_gain", "type": float, "doc": "Photomultiplier gain.", "default": None},
+        {
+            "name": "scan_line_rate",
+            "type": float,
+            "doc": (
+                "Lines imaged per second. This is also stored in /general/optophysiology but is kept "
+                "here as it is useful information for analysis, and so good to be stored w/ the actual data."
+             ),
+            "default": None,
+        },
+        {
+            "name": "exposure_time",
+            "type": float,
+            "doc": "Exposure time of the sample; often the inverse of the frequency.",
+            "default": None,
+        },
+        {
+            "name": "binning",
+            "type": (int, "uint"),
+            "doc": "Amount of pixels combined into 'bins'; could be 1, 2, 4, 8, etc.",
+            "default": None,
+        },
+        {
+            "name": "power",
+            "type": float,
+            "doc": "Power of the excitation in mW, if known.",
+            "default": None,
+        },
+        {
+            "name": "intensity",
+            "type": float,
+            "doc": "Intensity of the excitation in mW/mm^2, if known.",
+            "default": None,
+        },
+        *get_docval(
+            ImageSeries.__init__,
+            "external_file",
+            "starting_frame",
+            "bits_per_pixel",
+            "dimension",
+            "resolution",
+            "conversion",
+            "timestamps",
+            "starting_time",
+            "rate",
+            "comments",
+            "description",
+            "control",
+            "control_description",
+            "device",
+            "offset",
+        )
+    )
+    def __init__(self, **kwargs):
+        keys_to_set = (
+            "imaging_plane", "pmt_gain", "scan_line_rate", "exposure_time", "binning", "power", "intensity"
+        )
+        args_to_set = popargs_to_dict(keys_to_set, kwargs)
+        super().__init__(**kwargs)
+
+        if args_to_set["binning"] < 0:
+            raise ValueError(f"Binning value must be >= 0: {args_to_set['binning']}")
+        if isinstance(args_to_set["binning"], int):
+            args_to_set["binning"] = np.uint(args_to_set["binning"])
+
+        for key, val in args_to_set.items():
+            setattr(self, key, val)
+
+
 @register_class('TwoPhotonSeries', CORE_NAMESPACE)
 class TwoPhotonSeries(ImageSeries):
     """Image stack recorded over time from 2-photon microscope."""

src/pynwb/testing/make_test_files.py

@@ -2,6 +2,7 @@
 import numpy as np
 from pathlib import Path
 from pynwb import NWBFile, NWBHDF5IO, __version__, TimeSeries, get_class, load_namespaces
+from pynwb.file import Subject
 from pynwb.image import ImageSeries
 from pynwb.spec import NWBNamespaceBuilder, export_spec, NWBGroupSpec, NWBAttributeSpec
 
@@ -197,6 +198,23 @@ def _make_empty_with_extension():
     _write(test_name, nwbfile)
 
 
+def _make_subject_without_age_reference():
+    """Create a test file without a value for age_reference."""
+    nwbfile = NWBFile(session_description='ADDME',
+                      identifier='ADDME',
+                      session_start_time=datetime.now().astimezone())
+    subject = Subject(
+        age="P90D",
+        description="A rat",
+        subject_id="RAT123",
+    )
+
+    nwbfile.subject = subject
+
+    test_name = 'subject_no_age__reference'
+    _write(test_name, nwbfile)
+
+
 if __name__ == '__main__':
     # install these versions of PyNWB and run this script to generate new files
     # python src/pynwb/testing/make_test_files.py
@@ -221,3 +239,6 @@ def _make_empty_with_extension():
         _make_imageseries_non_external_format()
         _make_imageseries_nonmatch_starting_frame()
         _make_empty_with_extension()
+
+    if __version__ == "2.2.0":
+        _make_subject_without_age_reference()