Loading collections fails when derivatives is outside parent directory

[adelavega]

If layout is set up as such:
layout = BIDSLayout('/bids', derivatives='/fmriprep')

Then the following will fails
layout.get_collections()

because the TR for an image in the derivatives can't be found:

~/repos/pybids/bids/variables/io.py in load_variables(layout, types, levels, skip_empty, dataset, **kwargs)
     74         _kwargs = kwargs.copy()
     75         _kwargs.update(type_flags)
---> 76         dataset = _load_time_variables(layout, dataset, **_kwargs)
     77
     78     for t in ({'scans', 'sessions', 'participants'} & set(types)):

~/repos/pybids/bids/variables/io.py in _load_time_variables(layout, dataset, columns, scan_length, drop_na, events, physio, stim, regressors, skip_empty, **selectors)
    160
    161         tr = layout.get_metadata(img_f, suffix='bold', domains=domains,
--> 162                                  full_search=True)['RepetitionTime']
    163
    164         run = dataset.get_or_create_node('run', entities, image_file=img_f,

KeyError: 'RepetitionTime'

If I instead add a symlink to derivatives/fmriprep then it works.

This might be a more general issue of how meta-data is searched when the root folder differs.

[tyarkoni]

Can you try this in 0.8 (#369) and see what happens?

[adelavega]

Looks like the same issue in 0.8

[effigies]

Isn't this a feature? Aren't we supposed to be producing valid sidecars for derivatives? I'm pretty sure we had a discussion and metadata isn't supposed to bleed between datasets.

[adelavega]

That makes sense to me. In that case then its a bug that it does bleed between derivatives and the main dataset.

I assume fmriprep will soon (or already does) produce this sort of meta-data soon?

[effigies]

It should. It turns out this is a major pain to implement, though.

[tyarkoni]

@adelavega in 0.8, by default, get_collections will use scope='all', so it will try to retrieve data for both raw and derivatives. If you don't want to scan derivatives, pass scope='raw'. I think the failure is expected behavior in this case, as @effigies says. Or are you saying that it's still returning metadata from the raw project for derivatives files, independently of this failure?

[adelavega]

I used get_collections with default args in 0.8 on a layout with an fmriprep derivative outside the root. It failed, which it seems is expected.

I need to re-test what happens with a derivative folder that is actually inside the bids dir. You are saying it should fail again because meta-data should no longer bleed?

In this case, I'll have to temporarily modify the fmriprep datasets we've published to include TR metadata until fmriprep can produce it.

[tyarkoni]

I need to go look at what (if anything) the spec says about this. It may also require an overhaul of the way non-imaging data files are read in (there are other reasons to want to do that as well). For the time being, yeah, let's just work around the issue by putting a single .json file in the derivative folder root with the right RepetitionTime. There's a good chance that will end up being the prescribed fix anyway.

[tyarkoni]

I think in 0.9 (see sqlalchemy branch) metadata should no longer bleed between raw and derivative directories. Feel free to test and confirm.