[Bug]: 'Core' not a namespace

[rcpeene]

What happened?

I attempted to run PyNWB.NWBHDF5IO in a Jupyter notebook while remotely accessing nwb files from DANDI.
The code that yields an error is code from the docs. This snippet below

import fsspec
import pynwb
import h5py
from fsspec.implementations.cached import CachingFileSystem

# first, create a virtual filesystem based on the http protocol and use
# caching to save accessed data to RAM.
fs = CachingFileSystem(
    fs=fsspec.filesystem("http"),
    cache_storage="nwb-cache",  # Local folder for the cache
)

# next, open the file
with fs.open(file_url, "rb") as f:
    with h5py.File(f) as file:
        with pynwb.NWBHDF5IO(file=file, mode='r', load_namespaces=True) as io:
            nwbfile = io.read()

And then I received an error; 'core not a namespace'

Steps to Reproduce

1. Clone and and install PyNWB from the github repo as described here
2. Run pip install dandi and pip install fsspec requests aiohttp
3. Run a jupyter notebook locally on your machine.
4. Include and run a cell with the following snippet

from dandi import dandiapi
client = dandiapi.DandiAPIClient()   
my_dandiset = client.get_dandiset("000004")
file = my_dandiset.get_asset_by_path("sub-P18HMH/sub-P18HMH_ses-20080601_ecephys+image.nwb")
file_url = file.get_content_url(follow_redirects=1, strip_query=True)

5. Then include and run the snippet which caused the error shown in the above section

Traceback

The traceback is as follows

KeyError                                  Traceback (most recent call last)
Cell In [6], line 2
      1 import fsspec
----> 2 import pynwb
      3 import h5py
      4 from fsspec.implementations.cached import CachingFileSystem

File c:\users\carter.peene\desktop\projects\pynwb\src\pynwb\__init__.py:294
    290         kwargs['container'] = nwbfile
    291         super().export(**kwargs)
--> 294 from . import io as __io  # noqa: F401,E402
    295 from .core import NWBContainer, NWBData  # noqa: F401,E402
    296 from .base import TimeSeries, ProcessingModule  # noqa: F401,E402

File c:\users\carter.peene\desktop\projects\pynwb\src\pynwb\io\__init__.py:1
----> 1 from . import base as __base
      2 from . import core as __core
      3 from . import file as __file

File c:\users\carter.peene\desktop\projects\pynwb\src\pynwb\io\base.py:3
      1 from hdmf.build import LinkBuilder
----> 3 from .core import NWBContainerMapper
      4 from .. import register_map
      5 from ..base import TimeSeries, ProcessingModule

File c:\users\carter.peene\desktop\projects\pynwb\src\pynwb\io\core.py:9
      5 from hdmf.build import BuildManager
      7 from .. import register_map
----> 9 from pynwb.file import NWBFile
     10 from pynwb.core import NWBData, NWBContainer, ScratchData
     11 from pynwb.misc import Units

File c:\users\carter.peene\desktop\projects\pynwb\src\pynwb\file.py:14
     11 from hdmf.utils import docval, getargs, get_docval, popargs, popargs_to_dict, AllowPositional
     13 from . import register_class, CORE_NAMESPACE
---> 14 from .base import TimeSeries, ProcessingModule
     15 from .device import Device
     16 from .epoch import TimeIntervals

File c:\users\carter.peene\desktop\projects\pynwb\src\pynwb\base.py:12
      9 from hdmf.utils import get_data_shape
     11 from . import register_class, CORE_NAMESPACE
---> 12 from .core import NWBDataInterface, MultiContainerInterface, NWBData
     15 @register_class('ProcessingModule', CORE_NAMESPACE)
     16 class ProcessingModule(MultiContainerInterface):
     17     """ Processing module. This is a container for one or more containers
     18         that provide data at intermediate levels of analysis
     19 
     20         ProcessingModules should be created through calls to NWB.create_module().
     21         They should not be instantiated directly
     22     """

File c:\users\carter.peene\desktop\projects\pynwb\src\pynwb\core.py:51
     46             raise ValueError(error_msg)
     47         warn(error_msg)
     50 @register_class('NWBContainer', CORE_NAMESPACE)
---> 51 class NWBContainer(NWBMixin, Container):
     53     _fieldsname = '__nwbfields__'
     55     __nwbfields__ = tuple()

File c:\users\carter.peene\desktop\projects\pynwb\src\pynwb\__init__.py:130, in register_class.<locals>._dec(cls)
    129 def _dec(cls):
--> 130     __TYPE_MAP.register_container_type(namespace, neurodata_type, cls)
    131     return cls

File ~\AppData\Local\Programs\Python\Python39\lib\site-packages\hdmf\utils.py:645, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    643 def func_call(*args, **kwargs):
    644     pargs = _check_args(args, kwargs)
--> 645     return func(args[0], **pargs)

File ~\AppData\Local\Programs\Python\Python39\lib\site-packages\hdmf\build\manager.py:724, in TypeMap.register_container_type(self, **kwargs)
    722 ''' Map a container class to a data_type '''
    723 namespace, data_type, container_cls = getargs('namespace', 'data_type', 'container_cls', kwargs)
--> 724 spec = self.__ns_catalog.get_spec(namespace, data_type)  # make sure the spec exists
    725 self.__container_types.setdefault(namespace, dict())
    726 self.__container_types[namespace][data_type] = container_cls

File ~\AppData\Local\Programs\Python\Python39\lib\site-packages\hdmf\utils.py:645, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    643 def func_call(*args, **kwargs):
    644     pargs = _check_args(args, kwargs)
--> 645     return func(args[0], **pargs)

File ~\AppData\Local\Programs\Python\Python39\lib\site-packages\hdmf\spec\namespace.py:315, in NamespaceCatalog.get_spec(self, **kwargs)
    313 namespace, data_type = getargs('namespace', 'data_type', kwargs)
    314 if namespace not in self.__namespaces:
--> 315     raise KeyError("'%s' not a namespace" % namespace)
    316 return self.__namespaces[namespace].get_spec(data_type)

KeyError: "'core' not a namespace"

Operating System

Windows

Python Executable

Python

Python Version

3.9

Package Versions

aiohttp==3.8.3
aiosignal==1.2.0
anyio==3.6.1
appdirs==1.4.4
argon2-cffi==21.3.0
argon2-cffi-bindings==21.2.0
arrow==1.2.3
asciitree==0.3.3
asttokens==2.0.8
async-timeout==4.0.2
attrs==22.1.0
backcall==0.2.0
beautifulsoup4==4.11.1
bidsschematools==0.5.1
bleach==5.0.1
ccfwidget==0.5.3
certifi==2022.9.14
cffi==1.15.1
charset-normalizer==2.1.1
ci-info==0.3.0
click==8.1.3
click-didyoumean==0.3.0
cloudpickle==2.2.0
colorama==0.4.5
colorcet==3.0.0
contourpy==1.0.5
cycler==0.11.0
dandi==0.46.3
dandischema==0.7.1
dask==2022.9.1
debugpy==1.6.3
decorator==5.1.1
defusedxml==0.7.1
dnspython==2.2.1
email-validator==1.3.0
entrypoints==0.4
etelemetry==0.3.0
executing==1.0.0
fasteners==0.18
fastjsonschema==2.16.2
fonttools==4.37.3
fqdn==1.5.1
frozenlist==1.3.1
fscacher==0.2.0
fsspec==2022.8.2
h5py==3.7.0
hdmf==3.4.2
humanize==4.4.0
idna==3.4
imageio==2.22.0
importlib-metadata==4.12.0
interleave==0.2.1
ipydatawidgets==4.3.2
ipykernel==6.15.3
ipympl==0.9.2
ipysheet==0.5.0
ipython==8.5.0
ipython-genutils==0.2.0
ipytree==0.2.2
ipyvolume==0.5.2
ipywebrtc==0.6.0
ipywidgets==7.7.2
isoduration==20.11.0
itk-core==5.2.1.post1
itk-filtering==5.2.1.post1
itk-meshtopolydata==0.8.1
itk-numerics==5.2.1.post1
itkwidgets==0.32.3
jaraco.classes==3.2.2
jedi==0.18.1
Jinja2==3.1.2
JIT==0.0.1
joblib==1.2.0
jsonpointer==2.3
jsonschema==4.16.0
jupyter-core==4.11.1
jupyter-server==1.18.1
jupyter_client==7.3.5
jupyterlab-pygments==0.2.2
jupyterlab-widgets==1.1.1
keyring==23.9.3
keyrings.alt==4.2.0
kiwisolver==1.4.4
llvmlite==0.39.1
locket==1.0.0
lxml==4.9.1
MarkupSafe==2.1.1
matplotlib==3.6.0
matplotlib-inline==0.1.6
mistune==2.0.4
more-itertools==8.14.0
multidict==6.0.2
natsort==8.2.0
nbclient==0.6.8
nbconvert==7.0.0
nbformat==5.6.0
ndx-grayscalevolume==0.0.2
ndx-icephys-meta==0.1.0
ndx-spectrum==0.2.2
nest-asyncio==1.5.5
networkx==2.8.6
notebook==6.4.12
numba==0.56.2
numcodecs==0.10.2
numpy==1.21.5
nwbinspector==0.4.13
nwbwidgets==0.8.0
opencv-python==4.6.0.66
packaging==21.3
pandas==1.3.5
pandocfilters==1.5.0
param==1.12.2
parso==0.8.3
partd==1.3.0
pickleshare==0.7.5
Pillow==9.2.0
plotly==5.10.0
prometheus-client==0.14.1
prompt-toolkit==3.0.31
psutil==5.9.2
pure-eval==0.2.2
pycparser==2.21
pycryptodomex==3.15.0
pyct==0.4.8
pydantic==1.10.2
Pygments==2.13.0
-e git+https://github.com/NeurodataWithoutBorders/pynwb.git@6e1f131c32ead004271d143d00c6fee05dbd9ef6#egg=pynwb
pyout==0.7.2
pyparsing==3.0.9
pyrsistent==0.18.1
python-dateutil==2.8.2
pythreejs==2.4.1
pytz==2022.2.1
PyWavelets==1.4.1
pywin32==304
pywin32-ctypes==0.2.0
pywinpty==2.0.8
PyYAML==6.0
pyzmq==24.0.1
requests==2.28.1
rfc3339-validator==0.1.4
rfc3987==1.3.8
ruamel.yaml==0.17.21
ruamel.yaml.clib==0.2.6
scikit-image==0.19.3
scikit-learn==1.1.2
scipy==1.9.1
semantic-version==2.10.0
Send2Trash==1.8.0
six==1.16.0
sniffio==1.3.0
soupsieve==2.3.2.post1
stack-data==0.5.0
tenacity==8.1.0
terminado==0.15.0
threadpoolctl==3.1.0
tifffile==2022.8.12
tinycss2==1.1.1
toolz==0.12.0
tornado==6.2
tqdm==4.64.1
traitlets==5.4.0
traittypes==0.2.1
trimesh==3.15.2
typing_extensions==4.3.0
uri-template==1.2.0
urllib3==1.26.12
util-colleenjg==0.0.1
wcwidth==0.2.5
webcolors==1.12
webencodings==0.5.1
websocket-client==1.4.1
widgetsnbextension==3.6.1
xarray==2022.6.0
yarl==1.8.1
zarr==2.13.0
zipp==3.8.1
zstandard==0.18.0

Code of Conduct

• I agree to follow this project's Code of Conduct
• Have you checked the Contributing document?
• Have you ensured this bug was not already reported?

[rly]

To confirm, is this error from using the same dandiset described in the docs?

[rcpeene]

Ah, no its not! I neglected to include that.
It's dandiset_id 000004 and file sub-P18HMH/sub-P18HMH_ses-20080601_ecephys+image.nwb.
Will edit my instruction snippet to include this.

[bendichter]

@rcpeene

Can you please install pynwb using these instructions? You need to do a special command that includes submodules. It should work once you do that

[rcpeene]

Excellent. I think that fixed this issue, but I can't say verify this for certain because I am getting a new error now with the same code. Here's the traceback.

TypeError                                 Traceback (most recent call last)
Cell In [6], line 16
     14 with fs.open(file_url, "rb") as f:
     15     with h5py.File(f) as file:
---> 16         with pynwb.NWBHDF5IO(file=file, mode='r', load_namespaces=True) as io:
     17             nwbfile = io.read()

File ~\AppData\Local\Programs\Python\Python39\lib\site-packages\hdmf\utils.py:644, in docval.<locals>.dec.<locals>.func_call(*args, **kwargs)
    643 def func_call(*args, **kwargs):
--> 644     pargs = _check_args(args, kwargs)
    645     return func(args[0], **pargs)

File ~\AppData\Local\Programs\Python\Python39\lib\site-packages\hdmf\utils.py:637, in docval.<locals>.dec.<locals>._check_args(args, kwargs)
    635     if parse_err:
    636         msg = '%s: %s' % (func.__qualname__, ', '.join(parse_err))
--> 637         raise ExceptionType(msg)
    639 return parsed['args']

TypeError: NWBHDF5IO.__init__: missing argument 'path'

Is this me misusing the example code or a bug with the package?

[bendichter]

Did you do this?

$ git clone --recurse-submodules [email protected]:NeurodataWithoutBorders/pynwb.git
$ cd pynwb
$ pip install -r requirements.txt
$ pip install -e .

[oruebel]

TypeError: NWBHDF5IO.init: missing argument 'path'

I think you may need to update the version of HDMF you are using

[rcpeene]

I think the install step was interrupted for some reason and I missed it. These fixed the problem. Thanks!

[rcpeene]

As a point of curiosity, is installing via the github repo the only way to prevent this error? Are there resulting packaging issues when installing via pip? This is relevant for my usage of this tool in a Jupyter notebook where I'd like to try and make it's usage environment-agnostic.

[oruebel]

As a point of curiosity, is installing via the github repo the only way to prevent this error? Are there resulting packaging issues when installing via pip?

Installing via pip is fine

[rcpeene]

Seems as though the NWBHDF5IO.__init__: missing argument 'path' arises again when testing it from a pip installation.

[oruebel]

Seems as though the NWBHDF5IO.__init__: missing argument 'path' arises again when testing it from a pip installation.

Can you try again. There is a new release of HDMF which I believe includes the fix for this

[rly]

The new HDMF release does not fix this. But the latest dev branch of PyNWB does fix this. You have to install pynwb from source using the git clone command that @bendichter included above.