Allow `@` queries for metadata

[jona-sassenhagen]

Metadata-based querying of Epochs is very powerful. For example, these work:

import mne
path_kword = mne.datasets.kiloword.data_path() + '/kword_metadata-epo.fif'
kword = mne.read_epochs(path_kword)
kword["NumberOfLetters == 4 & ConsonantVowelProportion < .5"], kword["WORD == 'area'"]

But what doesn't work is pulling a variable from the current scope with @:

word = 'area'
kword.metadata.query("WORD == @word")

This works, in Pandas. But this errors:

kword["WORD == @word"]

Would be cool to have!

@choldgraf

[choldgraf]

ah yes, that would indeed be cool! +1

[jasmainak]

wouldn't this be redundant with PyBIDS ?

[jasmainak]

sorry my bad. Read too quickly

[dengemann]

Sounds good. If we already have pandas support in epochs it should be as complete as reasonably possible.

…

On Wed 8 Aug 2018 at 04:43, Mainak Jas ***@***.***> wrote: sorry my bad. Read too quickly — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub <#5400 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB0fiiGgmNWf_y3kl0kzrE3oPg-IL2Tnks5uOlBogaJpZM4VvI6m> .

[jona-sassenhagen]

wouldn't this be redundant with PyBIDS ?

I'm kinda wondering how you misread this now ...

[larsoner]

Probably living with too many open GitHub tabs or emails open, and clicking into one assuming it was the one you meant to respond to without actually checking. At least that's how it has happened to me in the past, too :)

[jona-sassenhagen]

I tried to understand how Pandas does it, and I don't. And a self-made workaround based on looking up stuff in locals probably wouldn't work either.

Why are we using pd.DataFrame.eval btw over .query?

[larsoner]

It was a while ago but I think we did have some discussion about which one to use. Maybe one is/was deprecated?

[larsoner]

... but looking back at #4414 I can't immediately tell. It does look like we do eval under the hood but in docs discuss query, so there is at least some inconsistency / doc bug, and possibly an actual bug for which we use.

I am not a pandas person so I can't comment meaningfully on which to use though.

[jona-sassenhagen]

I don't know ... I tried switching the eval out for a query and it didn't immediately work :)

@choldgraf ?

[jona-sassenhagen]

One of the points of eval vs. query in pandas is that eval can do assignments, whereas query can't - it just queries. Which feels like what we want here, actually.

Oh, and the scoping is different IIRC?

[choldgraf]

I'm pretty sure that they accomplish mostly the same thing, but that eval returns a boolean mask according to whatever the query was, while query returns the DataFrame object itself. So probably we use eval because we want that mask somewhere else in the code, I'm guessing.

[drammock]

... 2 years later, I tried to get this to work by manipulating the level kwarg (how far up the call stack pandas looks for @variables), as well as passing local_dict=globals() and a few other hackish things; nothing worked. Given that it's possible to generally replace use of @ with a fairly similar f-string nowadays (e.g., epochs['WORD in @words'] becomes epochs[f'WORD in {words}']) I think we can close this as YAGNI.