Support HED (Hierarchical Event Descriptors) for epoching

[VisLab]

Describe the new feature or enhancement

MNE-Python represents events by a set of discrete integer codes, and epoching functions mostly select epochs based on these codes.

I was interested in incorporating some epoching features based on queries of HED strings and was hoping to get some advice (and maybe a little help) on figuring out the best way to do that.

HED is incorporated as an event annotation mechanism in BIDS. We can easily produce a table of event number versus HED string annotations to start with. (Note these would be the original event positions not codes in a particular column, since HED annotations are usually assembled from information in several columns in a BIDS event file.) We can also select numbers based on a HED string query.

Describe your proposed implementation

I'd really appreciate some advice on where to start on this.

Describe possible alternatives

We could extend the Epoch class so that it would store the HED annotations as a function of sample number.

We could also do something with the Annotation class.

Perhaps better, we could write an alternative way of creating Epochs and as part of this attach a HED_assembled column to the Epochs metadata. We could then write a query interface.

Additional context

Note:
In a BIDS dataset the HED annotation for each event is assembled from a combination of the information in the events.tsv HED column and the information in the other columns based on any applicable sidecars. This information is combined during analysis to provide a single HED annotation of each event.

[welcome]

Hello! 👋 Thanks for opening your first issue here! ❤️ We will try to get back to you soon. 🚴🏽‍♂️

[agramfort]

can you give an example in Python code of manipulating HED? it has remained an obscure thing for me so far

…

Message ID: ***@***.***>

[VisLab]

Factorization based on HED tags is a good example. See https://www.hed-resources.org/en/latest/FileRemodelingTools.html#factor-hed-tags. The goal is to epoch based on a variety of criteria in a dataset-independent manner (This is where the HED tags come in). The remodeling tools shown in the link are one interface to the underlying HEDTools.

Consider the case where the HED tags for each event marker have been assembled into a string in the HED column.
The user creates a HED query string.

Example:

Sensory-event And Visual-presentation

A vector of 0's and 1's is produced of the same length as the original data indicating whether the query matches the corresponding HED string.

Here is some sample code that I've done a quick hack on, removing all of the error checking and other things. This code actually modifies the DataFrame coming in, which we never do. The underlying code is designed to process multiple queries at once, but this example only does one.

def make_factor(df, hed_strings, hed_schema, query_string, query_name):
    query = QueryParser(query_string)
    df_factor = DataFrame(0, index=range(len(hed_strings)), columns=[query_name])
    for index, next_item in enumerate(hed_strings):
        match = query.search(next_item)
        if match:
            df_factor.at[index, query_name] = 1
    df[query_name] = df_factor

[agramfort]

Can you share the dataframe and a self contained example I can just run on my machine?

[VisLab]

Attached is a zip file with a self-contained example.

quickExample.zip

This example just processes a single query to append a factor. We are doing some refactoring on the underlying infrastructure and I'd like to see how we might support HED querying in MNE-Python.

The goal is more flexible, dataset-independent querying.

P.S. You have to pip install hedtools.

[drammock]

I had a meeting today with @VisLab, @arnodelorme, @neuromechanist, @dungscout96, and Scott Makeig to discuss HED integration with MNE-Python and MNE-BIDS. Here is a summary of our tentative plan to hopefully make this happen; @mne-tools/mne-python-steering-committee please let us know what you think.

1. create a subclass of Annotations that has an additional object-level property hed_version, and an additional field hed_tag associated with each annotation. When created or modified, it will call out to https://github.com/hed-standard/hed-python to validate those fields.

2. make MNE-BIDS aware of this new HEDAnnotations class, so that when it writes Raw data to a BIDS dataset, it will extract the tags and put them in the appropriate place in events.json and/or the hed column in events.tsv, and also write the hed_version to the dataset's metadata.

3. make it so that when MNE-BIDS loads data from a BIDS dataset with HED info in the events.tsv table or events.json sidecar, it will insert that data into an HEDAnnotations object attached to the loaded Raw file.

4. make it easy to create epochs based on the HED tags in a HEDAnnotations object attached to a Raw. This last step we didn't discuss in much detail so this is a bit hand-wavy, but since we already have [ENH, MRG] Allow epoch construction from annotations #12311 it shouldn't be too difficult to triage based on annotation object type and extract the epoch condition names from hed_tag instead of description.

5. Possibly other stuff that @VisLab has in mind that we didn't manage to talk about because I bored her with nitty-gritty details about implementation :) Feel free to comment below to add them!

If y'all approve, @VisLab and I agreed to collaborate on a PR for item (1) --- she'll open the PR, I'll push commits creating the HEDAnnotations subclass, and then she'll add the validation code. Subsequent steps we haven't discussed who will do the work (volunteers welcome!).

[larsoner]

1-3 all sound good to me! I think the subclass is a good idea so that any HED-related methods etc. we might want end up in the HEDAnnotations subclass. And maybe we add a Annotations.to_hed(...) to allow people to convert their existing Annotations to HED if they want, for example if they've already used HED-compliant annotation description values. But we can always add this later!

4. make it easy to create epochs based on the HED tags in a HEDAnnotations object attached to a Raw. This last step we didn't discuss in much detail so this is a bit hand-wavy, but since we already have [ENH, MRG] Allow epoch construction from annotations [ENH, MRG] Allow epoch construction from annotations #12311 it shouldn't be too difficult to triage based on annotation object type and extract the epoch condition names from hed_tag instead of description.

Agreed, my initial thought is some new method perhaps. But we can prototype some options and see how they look.

[agramfort]

I tried your zip folder above but it fails with the current hedtools pypi package. What is your backward compat policy? Is the plan here to make hedtools a dependency of mne?

…

Message ID: ***@***.***>

[drammock]

What is your backward compat policy?

@VisLab should answer this, but IIRC at the meeting it was mentioned that back compatibility is expected to be preserved

Is the plan here to make hedtools a dependency of mne?

I was thinking optional dependency, using our _soft_import mechanism

[VisLab]

Sorry about the example -- it was using BIDS and not relevant to MNE-python. The backwards compatibility we discussed in the meeting was about the schema vocabulary. The HEDTools (python) was originally just a validator and the public interface to the other tools has changed slightly among versions of 0.x.x. There is still some movement on what we want as the final version of some of the other supporting tools. We hope to release a frozen version 1.0.0 in the second quarter of 2025.

I have attached a self-contained example. In Stage 1, the main concern is validation. The examples show how this is currently done. Stage 2 would be integrating HED into epoching. The MNE string search could be used, but we could also support the HED query mechanism. The example searches for 'Sensory-presentation'. This returns true for the items containing "Visual-presentation" and "Auditory-presentation" since these terms are children of "Sensory-presentation" in the HED schema.

There are various supporting tools such as a query validator which are not shown here. There are a lot of different types of HED matches (see HED search guide), but I think the 80% rule is that most users just use very simple searches.

hed_validation_and_query.zip

[drammock]

@VisLab whenever you're back in action after the winter holidays, feel free to open a near-empty draft-mode PR that just sticks a TODO comment right about here:

mne-python/mne/annotations.py

Line 759 in 47ea360

and then I'll start pushing commits to add the framework for the new HEDAnnotations subclass.