Bug in Matnwb: use nwb.units.getRow() to retrieve "waveforms" data

[jiumao2]

I wonder whether you have modified nwb.units.getRow() to get "waveforms" data?
I think I have set "waveforms_index" and "waveforms_index_index" right.
The "waveforms" data can't be retrieved with nwb.units.getRow(), while "spikes_time" can be retrieved normally.

This is what I have:

[jiumao2]

Sorry for putting this issue in wrong place

[oruebel]

@ln-vidrio

[bendichter]

@jjumao2 yes it's possible the getRow function does not work with multiple indexes. Good catch

[lawrence-mbf]

How is your colnames property set?

[lawrence-mbf]

Actually, it'd be helpful if you provided the snippet of code which populated the Unit table.

[jiumao2]

nwb.units = types.core.Units( ...
'colnames', {'spike_times', 'waveforms', 'waveforms_index', 'waveforms_index_index', 'unit_type','cluster_id','polytrode'}, ...
'description', ['units table, Definition for unit_type: ',r.Units.Definition{2}, ' ', r.Units.Definition{3}], ...
'id', types.hdmf_common.ElementIdentifiers( ...
'data', int64(0:length(spikes) - 1)), ...
'spike_times', spike_times_vector, ...
'spike_times_index', spike_times_index, ...
'waveforms', waveforms_vector, ...
'waveforms_index', waveforms_index_vector, ...
'waveforms_index_index', waveforms_index_index_vector, ...
'electrodes', electrode_table_region, ...
'electrodes_index', electrode_index, ...
'cluster_id', cluster_id, ...
'unit_type', unit_type, ...
'polytrode', polytrode);

waveforms_vector =

VectorData with properties:

description: 'None'
       data: [67852480×1 double]

waveforms_index_vector =

VectorIndex with properties:

     target: [1×1 types.untyped.ObjectView]
description: 'None'
       data: [1060195×1 uint32]

waveforms_index_index_vector =

VectorIndex with properties:

     target: [1×1 types.untyped.ObjectView]
description: 'None'
       data: [16×1 uint32]

16 spikes are detected. Each spike is dectected by one channel.
I have modified the file according to #302 but it doesn't help.

nwb.units.getRow(1)

ans =

1×8 table

  spike_times       spike_times_index     waveforms     waveforms_index     waveforms_index_index    unit_type    cluster_id    polytrode
________________    _________________    ___________    ________________    _____________________    _________    __________    _________

{42110×1 double}          42110          {[79.7143]}    {42110×1 uint32}            42110                1            1             0    

[lawrence-mbf]

@jiumao2 Can you pull from that branch and try getRow again?

[jiumao2]

Is it related to the "colname" or dimension of "waveforms"?
The following error emerged when trying my previous dataset.

Array indices must be positive integers or logical values.

Error in types.util.dynamictable.getRow>select (line 52)
column = colIndStack{end};

Error in types.util.dynamictable.getRow>select (line 77)
selected{iSelection} = select(...

Error in types.util.dynamictable.getRow (line 44)
row{i} = select(DynamicTable, indexNames, ind);

Error in types.hdmf_common.DynamicTable/getRow (line 121)
row = types.util.dynamictable.getRow(obj, id, varargin{:});

When I tried to form new dataset with identical codes, the function "util.create_indexed_column()" didn't work. I met the following error:

Error using types.util.correctType (line 60)
Value must be of signage uint

Error in types.util.checkDtype (line 111)
val = types.util.correctType(val, type);

Error in types.hdmf_common.VectorIndex/validate_data (line 36)
val = types.util.checkDtype('data', 'uint8', val);

Error in types.hdmf_common.Data/set.data (line 31)
obj.data = obj.validate_data(val);

Error in types.hdmf_common.Data (line 24)
obj.data = p.Results.data;

Error in types.hdmf_common.VectorData (line 15)
obj = [email protected]_common.Data(varargin{:});

Error in types.hdmf_common.VectorIndex (line 15)
obj = [email protected]_common.VectorData(varargin{:});

Error in util.create_indexed_column (line 44)
data_index = types.hdmf_common.VectorIndex( ...

Error in dataToNwb (line 334)
[spike_times_vector, spike_times_index] = util.create_indexed_column( ...

[lawrence-mbf]

If you have the dataset creation script that would be helpful as there are a multitude of ways to actually create dynamic tables. It currently looks like your waveforms dataset is not a VectorData object which is not expected.

I have fixed the util.create_indexed_column call as well if you wish to test that. This was due to a minor change to how index types are "shrinkwrapped".

[jiumao2]

waveforms_vector = types.hdmf_common.VectorData();
waveforms_vector.data = waveforms;
waveforms_vector.description = 'None';

target = types.untyped.ObjectView('/units/waveforms');

waveforms_index_vector = types.hdmf_common.VectorIndex();
waveforms_index_vector.data = (1:a)';
waveforms_index_vector.target = target;
waveforms_index_vector.description = 'None';

target2 = types.untyped.ObjectView('/units/waveforms_index');

waveforms_index_index_vector = types.hdmf_common.VectorIndex();
waveforms_index_index_vector.data = waveforms_index_index;
waveforms_index_index_vector.target = target2;
waveforms_index_index_vector.description = 'None';

nwb.units = types.core.Units( ...
    'colnames', {'spike_times', 'spike_times_index', 'waveforms', 'waveforms_index', 'waveforms_index_index','electrodes_index','unit_type','cluster_id','polytrode'}, ...
    'description', ['units table, Definition for unit_type: ',r.Units.Definition{2}, ' ', r.Units.Definition{3}], ...
    'id', types.hdmf_common.ElementIdentifiers( ...
    'data', int64(0:length(spikes) - 1)), ...
    'spike_times', spike_times_vector, ...
    'spike_times_index', spike_times_index, ...
    'waveforms', waveforms_vector, ...
    'waveforms_index', waveforms_index_vector, ...
    'waveforms_index_index', waveforms_index_index_vector, ...
    'electrodes', electrode_table_region, ...
    'electrodes_index', electrode_index, ...
    'cluster_id', cluster_id, ...
    'unit_type', unit_type, ...
    'polytrode', polytrode);

a: 1060195
waveforms: 1060195x64 double.
waveforms_index_index: 16x1 double (I have 16 units)

[lawrence-mbf]

So if I'm reading this right, you wish to divide the entire waveforms column into 16 parts evenly. In that case, you may not need waveforms_index_index (it is not a required property if you check the export function). So you can leave that property empty, omit it from colnames and use waveforms_index directly.

I'm also seeing that your data is a matrix and not an actual vector. That might be a bug on our end.

[lawrence-mbf]

@jiumao2 I have fixed multidimensional matrices.

Without manually debugging your ragged array indices, I'm not sure what other issue this could be. Would it be possible to reformat the code such that your code uses addRow instead? For instance, if your table only has 16 rows, you can do this:

dt = types.hdmf_common.DynamicTable('colnames', {'waveforms'});

matDims = [1060195 64];
numRows = 16;
segmentSize = floor(matDims(1) / numRows);
lastSeg = segmentSize + mod(matDims(1), segmentSize);

for i = 1:(numRows - 1)
    dt.addRow('waveforms', repmat(i, [segmentSize matDims(2)]));
end

dt.addRow('waveforms', repmat(numRows, [lastSeg matDims(2)]));

Not dynamically generating the data obviously.

[lawrence-mbf]

Closed as stale