Error reading example datasets with nwbRead

[ftadel]

I'm trying to use matnwb to read EEG-like signals from .nwb files in Brainstorm (https://neuroimage.usc.edu/brainstorm).
One of our former collaborators used matnwb in early 2019 to import ECoG signals, but I can't use any of his work with more recent versions of matnwb...

Since we have users who are interested in processing NWB recordings in Brainstorm, I am taking over this work. We started a conversation on our user forum: https://neuroimage.usc.edu/forums/t/error-opening-nwb-files/21025/11

I tried to start fresh with the recent versions of the library (current github master, and v2.2.5.0) and the example datasets available on the NWB website (https://www.nwb.org/example-datasets/) and other websites (https://gui.dandiarchive.org/#/dandiset/000020/draft), but I'm facing many issues with almost all the datasets.
Below: a list of example files and the errors I obtain with the two versions of the library I tested.

Installation of the Github master branch (https://github.com/NeurodataWithoutBorders/matnwb/archive/master.zip):

>> cd C:\Users\franc\Downloads\matnwb-master
>> generateCore
Found "matnwb" root directory at: C:\Users\franc\Downloads\matnwb-master.
Found "namespaces" directory at: C:\Users\franc\Downloads\matnwb-master\namespaces.
Found "matnwb" root directory at: C:\Users\franc\Downloads\matnwb-master.
Found "namespaces" directory at: C:\Users\franc\Downloads\matnwb-master\namespaces.
Found "matnwb" root directory at: C:\Users\franc\Downloads\matnwb-master.
Found "namespaces" directory at: C:\Users\franc\Downloads\matnwb-master\namespaces.
>>

Installation of the v2.2.5.0 version (https://github.com/NeurodataWithoutBorders/matnwb/archive/v2.2.5.0.tar.gz):

>> cd C:\Users\franc\Downloads\matnwb-2.2.5.0
>> generateCore
>> 

nwb_an041_20140821_vM1_180um.nwb

matnwb-2.2.5.0:

>> nwb = nwbRead('C:\Work\RawData\Test\NWB\nwb_an041_20140821_vM1_180um.nwb')
Error using types.util.checkUnset (line 13)
Unexpected properties {source}.

Your schema version may be incompatible with the file.  Consider checking the schema version of the file with
`util.getSchemaVersion(filename)` and comparing with the YAML namespace version present in nwb-schema/core/nwb.namespace.yaml

Error in types.core.TimeSeries (line 82)
            types.util.checkUnset(obj, unique(varargin(1:2:end)));

Error in io.parseGroup (line 85)
    parsed = eval([Type.typename '(kwargs{:})']);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in nwbRead (line 33)
nwb = io.parseGroup(filename, h5info(filename), Blacklist);
 
>> 

>> nwb = util.getSchemaVersion('C:\Work\RawData\Test\NWB\nwb_an041_20140821_vM1_180um.nwb')
Error using hdf5lib2
The HDF5 library encountered an error and produced the following stack trace information:

    H5O_attr_open_by_name    can't locate attribute: 'nwb_version'
    H5Aopen                  unable to load attribute info from object header for attribute: 'nwb_version'

Error in H5A.open (line 34)
id = H5ML.hdf5lib2('H5Aopen', obj_id, attr_name, acpl_id);

Error in util.getSchemaVersion (line 3)
aid = H5A.open(fid, 'nwb_version');

matnwb-master:

>> nwb = nwbRead('C:\Work\RawData\Test\NWB\nwb_an041_20140821_vM1_180um.nwb')
Error using types.util.checkUnset (line 13)
Unexpected properties {help source}.

Your schema version may be incompatible with the file.  Consider checking the schema version of the file
with `util.getSchemaVersion(filename)` and comparing with the YAML namespace version present in
nwb-schema/core/nwb.namespace.yaml

Error in types.core.TimeSeries (line 82)
            types.util.checkUnset(obj, unique(varargin(1:2:end)));

Error in io.parseGroup (line 85)
    parsed = eval([Type.typename '(kwargs{:})']);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in nwbRead (line 33)
nwb = io.parseGroup(filename, h5info(filename), Blacklist);
 
>> 

>> nwb = util.getSchemaVersion('C:\Work\RawData\Test\NWB\nwb_an041_20140821_vM1_180um.nwb')
Error using hdf5lib2
The HDF5 library encountered an error and produced the following stack trace information:

    H5O_attr_open_by_name    can't locate attribute: 'nwb_version'
    H5Aopen                  unable to load attribute info from object header for attribute: 'nwb_version'

Error in H5A.open (line 34)
id = H5ML.hdf5lib2('H5Aopen', obj_id, attr_name, acpl_id);

Error in util.getSchemaVersion (line 3)
aid = H5A.open(fid, 'nwb_version');

P9HMH_NOID5.nwb

matnwb-master:

>> nwb = nwbRead('C:\Work\RawData\Test\NWB\P9HMH_NOID5.nwb')
Unable to resolve the name types.core.VectorData.

Error in io.parseDataset (line 72)
    parsed = eval([Type.typename '(kwargs{:})']);

Error in io.parseGroup (line 22)
    dataset = io.parseDataset(filename, datasetInfo, fullPath, Blacklist);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in nwbRead (line 33)
nwb = io.parseGroup(filename, h5info(filename), Blacklist);
 
>> 

matnwb-2.2.5.0:

>> nwb = nwbRead('C:\Work\RawData\Test\NWB\P9HMH_NOID5.nwb')
Error using eval
Unrecognized function or variable 'types.core.VectorData'.

Error in io.parseDataset (line 72)
    parsed = eval([Type.typename '(kwargs{:})']);

Error in io.parseGroup (line 22)
    dataset = io.parseDataset(filename, datasetInfo, fullPath, Blacklist);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in nwbRead (line 33)
nwb = io.parseGroup(filename, h5info(filename), Blacklist);

Steinmetz2019_Cori_2016-12-14.nwb

matnwb-2.2.5.0:

>> nwb = nwbRead('C:\Work\RawData\Test\NWB\Steinmetz2019_Cori_2016-12-14.nwb')
The class 'types.hdmf_common.Container' is not a superclass of class 'types.core.NWBContainer', as required to invoke a superclass
constructor or method.

Error in types.core.NWBContainer (line 10)
        obj = [email protected]_common.Container(varargin{:});

Error in types.core.NWBDataInterface (line 10)
        obj = [email protected](varargin{:});

Error in types.core.TimeSeries (line 45)
        obj = [email protected](varargin{:});

Error in io.parseGroup (line 85)
    parsed = eval([Type.typename '(kwargs{:})']);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in nwbRead (line 33)
nwb = io.parseGroup(filename, h5info(filename), Blacklist);
 
>> 

matnwb-master: WORKS

sub-626194774_ses-637919731_icephys.nwb

matnwb-2.2.5.0:

>> nwb = nwbRead('C:\Work\RawData\Test\NWB\sub-626194774_ses-637919731_icephys.nwb')
Dot indexing is not supported for variables of this type.

Error in file.fillExport>traverseRaw (line 120)
        attrmatch = strcmp({raw.attributes.name}, propname);

Error in file.fillExport>traverseRaw (line 111)
                    res = traverseRaw(suffix, raw.datasets(i));

Error in file.fillExport (line 23)
    pathProps = traverseRaw(pnm, raw);

Error in file.fillClass (line 93)
exporterFcns = file.fillExport(nonInherited, class, depnm);

Error in file.writeNamespace (line 22)
        fwrite(fid, file.fillClass(className, Namespace, processed, ...

Error in nwbRead>tryWriteSpec (line 103)
    file.writeNamespace(namespaceName);

Error in nwbRead>generateSpec (line 91)
    if ~tryWriteSpec(name)

Error in nwbRead>checkEmbeddedSpec (line 44)
    generateSpec(fid, h5info(filename, specLocation));

Error in nwbRead (line 27)
    specLocation = checkEmbeddedSpec(filename);
 
>> 

matnwb-master:

>> nwb = nwbRead('C:\Work\RawData\Test\NWB\sub-626194774_ses-637919731_icephys.nwb')
Found "matnwb" root directory at: C:\Users\franc\Downloads\matnwb-master.
Found "namespaces" directory at: C:\Users\franc\Downloads\matnwb-master\namespaces.
Found "matnwb" root directory at: C:\Users\franc\Downloads\matnwb-master.
Found "namespaces" directory at: C:\Users\franc\Downloads\matnwb-master\namespaces.
Found "matnwb" root directory at: C:\Users\franc\Downloads\matnwb-master.
Found "namespaces" directory at: C:\Users\franc\Downloads\matnwb-master\namespaces.
Found "matnwb" root directory at: C:\Users\franc\Downloads\matnwb-master.
Found "namespaces" directory at: C:\Users\franc\Downloads\matnwb-master\namespaces.
Found "matnwb" root directory at: C:\Users\franc\Downloads\matnwb-master.
Found "namespaces" directory at: C:\Users\franc\Downloads\matnwb-master\namespaces.
Found "matnwb" root directory at: C:\Users\franc\Downloads\matnwb-master.
Found "namespaces" directory at: C:\Users\franc\Downloads\matnwb-master\namespaces.
Array indices must be positive integers or logical values.

Error in file.fillExport (line 24)
    prop = pathProps{end};

Error in file.fillClass (line 93)
exporterFcns = file.fillExport(nonInherited, class, depnm);

Error in file.writeNamespace (line 22)
        fwrite(fid, file.fillClass(className, Namespace, processed, ...

Error in nwbRead>tryWriteSpec (line 103)
    file.writeNamespace(namespaceName);

Error in nwbRead>generateSpec (line 91)
    if ~tryWriteSpec(name)

Error in nwbRead>checkEmbeddedSpec (line 44)
    generateSpec(fid, h5info(filename, specLocation));

Error in nwbRead (line 27)
    specLocation = checkEmbeddedSpec(filename);
 
>> 

YutaMouse20-140328.nwb

matnwb-2.2.5.0:

>> nwb = nwbRead('C:\Work\RawData\Test\NWB\YutaMouse20-140328.nwb')
Error using hdf5lib2
The filename specified was either not found on the MATLAB path or it contains unsupported characters.

Error in H5F.open (line 76)
file_id = H5ML.hdf5lib2('H5Fopen', filename, flags, fapl);

Error in nwbRead>checkEmbeddedSpec (line 40)
    fid = H5F.open(filename);

Error in nwbRead (line 27)
    specLocation = checkEmbeddedSpec(filename);
 
>> 
>> exist('C:\Work\RawData\Test\NWB\YutaMouse20-140328.nwb')

ans =

     2

matnwb-master:

>> nwb = nwbRead('C:\Work\RawData\Test\NWB\YutaMouse20-140328.nwb')
Error using hdf5lib2
The filename specified was either not found on the MATLAB path or it contains unsupported characters.

Error in H5F.open (line 76)
file_id = H5ML.hdf5lib2('H5Fopen', filename, flags, fapl);

Error in nwbRead>checkEmbeddedSpec (line 40)
    fid = H5F.open(filename);

Error in nwbRead (line 27)
    specLocation = checkEmbeddedSpec(filename);
 
>> exist('C:\Work\RawData\Test\NWB\YutaMouse20-140328.nwb')

ans =

     2

[lawrence-mbf]

I would advise re-downloading the tip of the master branch as a few minor fixes were just added in.

Regarding nwb_an041_20140821_vM1_180um.nwb:
Run util.getSchemaVersion('nwb_an041_20140821_vM1_180um.nwb'). This will print out the schema version you will need. I can assist in finding the correct schema you should be using for this file. This error comes up because there is no embedded specification in this particular file.

P9HMH_NOID5.nwb looks like a specification problem but I may need to take a closer look at the file itself before coming to any conclusions.

I will need to look at sub-626194774_ses-637919731_icephys.nwb to see what the embedded spec actually says here.

For YutaMouse20-140328.nwb it looks like the HDF5 library MATLAB is using does not like dashes in its filename. Does removing the dash in the filename fix the issue?

[ftadel]

Thank you replying so quickly!

I would advise re-downloading the tip of the master branch as a few minor fixes were just added in.

Some more warnings, but same errors.

nwb_an041_20140821_vM1_180um.nwb

>> generateCore
>> nwb = nwbRead('C:\Work\RawData\Test\NWB\nwb_an041_20140821_vM1_180um.nwb')
Warning: No embedded specification found. Ensure your environment is accurate to what is stored in this NWB
file.
To disable this warning run: `warning('off', 'MatNWB:NWBRead:NoSpec')` 
> In nwbRead>checkEmbeddedSpec (line 52)
In nwbRead (line 27) 
Error using types.util.checkUnset (line 13)
Unexpected properties {help source}.

Your schema version may be incompatible with the file.  Consider checking the schema version of the file
with `util.getSchemaVersion(filename)` and comparing with the YAML namespace version present in
nwb-schema/core/nwb.namespace.yaml

Error in types.core.TimeSeries (line 82)
            types.util.checkUnset(obj, unique(varargin(1:2:end)));

Error in io.parseGroup (line 85)
    parsed = eval([Type.typename '(kwargs{:})']);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in nwbRead (line 33)
nwb = io.parseGroup(filename, h5info(filename), Blacklist);
 
>> util.getSchemaVersion('C:\Work\RawData\Test\NWB\nwb_an041_20140821_vM1_180um.nwb')
Error using hdf5lib2
The HDF5 library encountered an error and produced the following stack trace information:

    H5O_attr_open_by_name    can't locate attribute: 'nwb_version'
    H5Aopen                  unable to load attribute info from object header for attribute: 'nwb_version'

Error in H5A.open (line 34)
id = H5ML.hdf5lib2('H5Aopen', obj_id, attr_name, acpl_id);

Error in util.getSchemaVersion (line 3)
aid = H5A.open(fid, 'nwb_version');
 
>> 

P9HMH_NOID5.nwb

>> nwb = nwbRead('C:\Work\RawData\Test\NWB\P9HMH_NOID5.nwb')
Warning: No embedded specification found. Ensure your environment is accurate to what is stored in this
NWB file.
To disable this warning run: `warning('off', 'MatNWB:NWBRead:NoSpec')` 
> In nwbRead>checkEmbeddedSpec (line 52)
In nwbRead (line 27) 
Unable to resolve the name types.core.VectorData.

Error in io.parseDataset (line 72)
    parsed = eval([Type.typename '(kwargs{:})']);

Error in io.parseGroup (line 22)
    dataset = io.parseDataset(filename, datasetInfo, fullPath, Blacklist);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in io.parseGroup (line 38)
    subg = io.parseGroup(filename, group, Blacklist);

Error in nwbRead (line 33)
nwb = io.parseGroup(filename, h5info(filename), Blacklist);
 
>> 

sub-626194774_ses-637919731_icephys.nwb

>> nwb = nwbRead('C:\Work\RawData\Test\NWB\sub-626194774_ses-637919731_icephys.nwb')

Array indices must be positive integers or logical values.

Error in file.fillExport (line 24)
    prop = pathProps{end};

Error in file.fillClass (line 93)
exporterFcns = file.fillExport(nonInherited, class, depnm);

Error in file.writeNamespace (line 22)
        fwrite(fid, file.fillClass(className, Namespace, processed, ...

Error in nwbRead>tryWriteSpec (line 106)
    file.writeNamespace(namespaceName);

Error in nwbRead>generateSpec (line 94)
    if ~tryWriteSpec(name)

Error in nwbRead>checkEmbeddedSpec (line 44)
    generateSpec(fid, h5info(filename, specLocation));

Error in nwbRead (line 27)
    specLocation = checkEmbeddedSpec(filename);

util.getSchemaVersion('nwb_an041_20140821_vM1_180um.nwb')

>> util.getSchemaVersion('C:\Work\RawData\Test\NWB\nwb_an041_20140821_vM1_180um.nwb')
Error using hdf5lib2
The HDF5 library encountered an error and produced the following stack trace information:

    H5O_attr_open_by_name    can't locate attribute: 'nwb_version'
    H5Aopen                  unable to load attribute info from object header for attribute: 'nwb_version'

Error in H5A.open (line 34)
id = H5ML.hdf5lib2('H5Aopen', obj_id, attr_name, acpl_id);

Error in util.getSchemaVersion (line 3)
aid = H5A.open(fid, 'nwb_version');

For YutaMouse20-140328.nwb it looks like the HDF5 library MATLAB is using does not like dashes in its filename. Does removing the dash in the filename fix the issue?

No...

>> nwb = nwbRead('C:\Work\RawData\Test\NWB\YutaMouse20140328.nwb')
Error using hdf5lib2
The filename specified was either not found on the MATLAB path or it contains unsupported characters.

Error in H5F.open (line 76)
file_id = H5ML.hdf5lib2('H5Fopen', filename, flags, fapl);

Error in nwbRead>checkEmbeddedSpec (line 40)
    fid = H5F.open(filename);

Error in nwbRead (line 27)
    specLocation = checkEmbeddedSpec(filename);

[lawrence-mbf]

It looks like nwb_an041_20140821_vM1_180um.nwb and P9HMH_NOID5.nwb are both using older versions of the schema that is packaged with the current tip of master. Call util.getSchemaVersion on these as well and we'll have to resolve these missing dependencies manually.

sub-626194774_ses-637919731_icephys.nwb is weird though. It may actually be a NWB 1.0 file. Can you call nwbTree(filename) on that file and tell me if you can find some equivalent to "nwb_version". It should be a string attribute in the file's root group.

For YutaMouse20-140328.nwb, is that full path to the file obfuscated or is it the real file path? This may actually be a bug with the HDF5 library bundled with MATLAB otherwise.

[ftadel]

It looks like nwb_an041_20140821_vM1_180um.nwb and P9HMH_NOID5.nwb are both using older versions of the schema that is packaged with the current tip of master. Call util.getSchemaVersion on these as well and we'll have to resolve these missing dependencies manually.

>> util.getSchemaVersion('C:\Work\RawData\Test\NWB\nwb_an041_20140821_vM1_180um.nwb')
Error using hdf5lib2
The HDF5 library encountered an error and produced the following stack trace information:

    H5O_attr_open_by_name    can't locate attribute: 'nwb_version'
    H5Aopen                  unable to load attribute info from object header for attribute: 'nwb_version'

Error in H5A.open (line 34)
id = H5ML.hdf5lib2('H5Aopen', obj_id, attr_name, acpl_id);

Error in util.getSchemaVersion (line 3)
aid = H5A.open(fid, 'nwb_version');

>> util.getSchemaVersion('C:\Work\RawData\Test\NWB\P9HMH_NOID5.nwb')

ans =

    '2.1.0'

sub-626194774_ses-637919731_icephys.nwb is weird though. It may actually be a NWB 1.0 file. Can you call nwbTree(filename) on that file and tell me if you can find some equivalent to "nwb_version". It should be a string attribute in the file's root group.

For all the .nwb files I tested, I get an empty window both with Matlab 2020a and 2020b.

util.nwbTree('C:\Work\RawData\Test\NWB\sub-626194774_ses-637919731_icephys.nwb')

For YutaMouse20-140328.nwb, is that full path to the file obfuscated or is it the real file path?

This is the absolute file path on my hard drive. If I rename it to test.nwb and cd to that folder, I get the same error.

>> addpath C:\Users\franc\Downloads\matnwb-master
>> cd C:\Work\RawData\Test\NWB
>> nwb = nwbRead('test.nwb')
Error using hdf5lib2
The filename specified was either not found on the MATLAB path or it contains unsupported characters.

Error in H5F.open (line 76)
file_id = H5ML.hdf5lib2('H5Fopen', filename, flags, fapl);

Error in nwbRead>checkEmbeddedSpec (line 40)
    fid = H5F.open(filename);

Error in nwbRead (line 27)
    specLocation = checkEmbeddedSpec(filename);

This may actually be a bug with the HDF5 library bundled with MATLAB otherwise.

These are all files from the datasets distributed as examples from the NWB website, you can test them easily.
It would be very helpful if all the resources listed on the website were actually working, even if they are less examples.

Are there other websites from which we can download example datasets that are easy to work with using the current version of the library?

[lawrence-mbf]

P9HMH_NOID5.nwb is using schema 2.1.0.
To downgrade the current schema to this version:

1. download the schema from here: https://github.com/NeurodataWithoutBorders/nwb-schema/releases/tag/2.1.0
2. Unzip the Source file and copy the contents of nwb-schema-2.1.0/core into nwb-schema/core in the MatNWB installation directory.
3. call generateCore again. You should then be able to call nwbRead(filename, 'ignorecache') to ignore the embedded cache and read the file.

YutaMouse20-140328.nwb appears to be corrupted and unreadable even in HDFViewer. I will check YutaMouse20-140327.nwb. This may also be the case with nwb_an041_20140821_vM1_180um.nwb so I will also take a look there.

Which dataset from the NWB website is sub-626194774_ses-637919731_icephys.nwb from?

[lawrence-mbf]

YutaMouse20-140327.nwb works. I suggest using that one but you do need to downgrade the schema to version 2.0.1 using the same instructions above with P9HMH_NOID5.nwb. You can find that schema version here: https://github.com/NeurodataWithoutBorders/nwb-schema/releases/tag/2.0.1

[lawrence-mbf]

You can also find much more up-to-date datasets at https://gui.dandiarchive.org/#/. The ones on this page need updating. The team has been notified of this.

[ftadel]

Thank you for your prompt help and suggestions!

However, my goal is not really to read these files... I was hoping to be able to simply add the matnwb library to the Brainstorm environment (downloaded automatically from github when needed) to offer a native support for the NWB file format to the Brainstorm users, as it was accepted as one of the reference file format in the BIDS-EEG specifications.

But it looks a lot more complicated than expected. Apparently it will not be possible to include a version of the library that will be able to read all the .nwb files, it will always require some customized manipulations, and navigation between versions...

Another problem is that is doesn't seem to be clear how EEG/LFP/SEEG/ECOG signals should be stored in the NWB ontology. The various example files I could load in MATLAB do not organize the data in the same way. It doesn't seem easy to write code that could automatically find all the information it needs. This would require an important amount of work to supervise this interactive import of EEG signals from .nwb files into Brainstorm, and then to document it.

Our development resources on the Brainstorm project are limited and we have many projects to lead simultaneously. Unfortunately, we won't be able to invest time in writing supervision tools to manage multiple versions of the matnwb library or the schemas.
At the present time, it is not clear to me whether the matnwb library or the NWB file format are ready for a plug-and-play use from an EEG-processing software environment. Maybe I should wait for an extra year and try again later?

What are your plans for NWB/matnwb in the near future?
Will these problems of schema compatibility be handled more smoothly?
Will you provide more strict specifications/examples/tutorials for EEG/SEEG/ECoG/LFP?
Can you provide help (eg. development time) for integrating your library into external software?

Unless you have better suggestions, I will remove the NWB support currently available in the Brainstorm distribution: it was developed for one specific dataset by a former PhD student of our group (his work is mentioned on your website: https://www.nwb.org/tools/), but this code is not working with any other public dataset available using .nwb files. It needs more work than what we can provide at the moment to be maintained properly.

[ftadel]

(I will not keep on investigating this immediately, please close this issue if you think it is not relevant for the immediate development of the matnwb library)

[lawrence-mbf]

I apologize for the poor first impressions that the NWB website gave you. pynwb and MatNWB have introduced the embedded specification feature a year (or so) ago which should allow anyone to simply nwbRead a file without having to set up their schema environment first.

As you've seen, this is not universal and does not work for old files but all files generated in the past year (and maybe older) should not require this environment setup. We expect this new format to phase out the old over time. I'd expect many of the files in the DANDI archive should have an embedded specification.

Regarding ontologies I will have to refer to @bendichter for more answers on integration.

I can only speak for myself but I am limited in the time I can spend developing for MatNWB as well. I can answer questions, help with bugs, or review PRs. Any more will require some negotiation.

[ftadel]

I will wait for a bit longer until this settles down then before looking into an integration with Brainstorm then (probably until the next post about NWB on our user forum).
Thanks

[bendichter]

@ftadel Thank you for bringing these issues to our attention.

We agree that MatNWB should offer seamless user experience for data reading, and that being required to find specific versions of schema would not be appropriate for integration into analysis and visualization tools like Brainstorm. There has been a lull in funding for this API, and it has been challenging to accommodate new versions of NWB file supporting old files. There are promising avenues for continued funding in the next month or so, and once one of those kicks in this issue will be a top priority.

As Lawrence mentioned, we have a new mechanism in place for handling the schema as it improves, but it will not work for data that was produced before this mechanism was put in place. (aside: @ln-vidrio , to solve this issues for older data, would it be possible to include the nwb schema repo as a sub-repo and dynamically use git to check out the schema version listed in the file? If there is potential in this sort of solution, maybe we can open a separate issue.)

It is important to me that we maintain the infrastructure that connects NWB to Brainstorm. @ftadel you raise an important point about the flexibility of NWB making it difficult to predict the form of the file. A recent feature, nwb.searchFor, might help with this. This has been added since @mpompolas's work, and helps the user identify the location of data that has a specific type. We are also working on supporting ontologies (hdmf-dev/hdmf-common-schema#20), but that will take some time. I think in the meantime, it might make sense for Brainstorm to offer an importer where a user can navigate through the NWB file and select the appropriate dataset to import, guided by the new searchFor tool. I understand that this would take substantial development effort, so it might be something that we could look for funding to support.

Again, we want to thank you for raising these issues to our attention. If you are having trouble reading published data using this API, then it is surely not meeting the needs of our users who generally have less programming experience. @ln-vidrio is doing excellent work here, but this lull in funding has created a bit of a gap. You came in just at the right time to reveal that gap, which will help us steer our efforts going forward. We would be interested in inviting you to try this API again once we have had some time to address these issues.

[lawrence-mbf]

@bendichter Regarding the schema issue, I think you can add a dynamic version selector but you'll need the following requirements:

1. nwb-schema tags need to be accurate to schema version.
2. The URL for the directories should be resistant to change.

This isn't universal (i.e. 2.0a is a valid NWB schema version but doesn't exist as a tag) but it would be helpful for the regular case as a fallback.

[mpompolas]

Let me add here for completeness that there is also the Java issue we encountered before:
#124

GenerateCore can not be called while Brainstorm is running.
So even if you download it through nwbRead, if generateCore needs to run for the right schema, you'll get Java problems.

The Brainstorm side workaround for this was described in:
brainstorm-tools/brainstorm3#181

However now that there is a need for support of multiple versions, this solution is no longer viable.

Otherwise, a cache of all the working schemas should exist on the Brainstorm side, and call the right one when the file is imported/read.

[bendichter]

about the YutaMouse20-140328.nwb dataset, this one was corrupted in transfer.

There are one or two more in that dataset that were corrupted. All the larger datasets are working as expected, but this is a common problem as no one wants to download a 10GB file just to test code. Actually, I'd recommend downloading this data from DANDI here https://dandiarchive.org/dandiset/000003/draft where all of the corrupted data has been fixed. This link, and many of the links on the page, need to be updated. I'll fix that.

[alexdiz]

Hi,
I have the exact same error trying to use electrophysiology data from Allen Atlas https://celltypes.brain-map.org/experiment/electrophysiology/584254833

Error using hdf5lib2
The HDF5 library encountered an error and produced the following stack trace information:

H5O_attr_open_by_name    can't locate attribute: 'nwb_version'
H5Aopen                  unable to load attribute info from object header for attribute: 'nwb_version'

Error in H5A.open (line 34)
id = H5ML.hdf5lib2('H5Aopen', obj_id, attr_name, acpl_id);

Error in util.getSchemaVersion (line 3)
aid = H5A.open(fid, 'nwb_version');

Error in nwbRead (line 47)
generateCore(util.getSchemaVersion(filename), 'savedir', saveDir);

And I can't get any information about schema version using util.func like util.getSchemaVersion or
util.nwbTree which gives me the same error message

have you any clues ?
Thanks
Alex

[lawrence-mbf]

Hi @alexdiz,

In your case, this looks like an older version of NWB that did not encode a nwb_version (or it's not a NWB file at all). This is required by the NWB standard now so the only solution (sans rewriting the data) would be to get an old MatNWB instance to open it. Otherwise you will probably be limited to using the HDF5 library calls in MATLAB instead.

[lawrence-mbf]

Cached spec has been completed for a while.