Issue with study stats

[arnodelorme]

I'm exploring STUDY statistics for the first time in a 2x2 factorial design. I have sequentially compared "Compute 1st independent variable..." and "Compute 2nd independent variable...", using EEGLAB's permutation/Bonferroni option (n=1000, threshold = .05) and using Fieldtrip statistics (montecarlo, cluster correction, n=1000, threshold = .05).
The results are almost impossible to reconcile, unless what the EEGLAB statistics consider the 1st independent variable is the 2nd one for the Fieldtrip statistics option and vice versa.

[arnodelorme]

I am confused, because the fieldtrip option cannot compute 2x2 designs, only 1xm designs.

[KrisBaetens]

I understand, because I might not have expressed myself very clearly :). So, I have a 2x2 design, manipulating Go/NoGo, and alcohol versus non-alcohol related context. In all the below examples, I show scalp maps in the 300-500 ms. As one can readily observe, there is a robust Go/NoGo effect, and maybe, maybe not, a subtle influence of alcohol context. So, running:

EEGLAB stats 1st factor (only), bonferroni, threshold .01, n = 1000, you get:

So, no effect here. Alright, so I take the title of the right plot to mean "(Collapsed across) Go/NoGo", there not being much of an effect of alcohol context. That would be a counter-intuitive labeling of the figure imho (it would be more straightforward if GoNoGo displays the effect of the factor Go/NoGo), but it is not impossible.

Fieldtrip stats 1st factor (only) Fieldtrip (monte carlo, .01, n = 1000, cluster correction), one gets:

At first I thought, okay, the cluster correction is probably more lenient, makes sense, here an effect of alcohol is detected. So, progressing with otherwise identical parameters, but moving on to the 2nd independent (only),

EEGLAB 2nd factor(only):

So, as is immediately apparent, this looks very similar to the fieldtrip results for the first factor, previous plot.

Fieldtrip 2nd factor (only):

There is an effect here, which is not present in the EEGLAB stats for the 1st independent. However, if we make the threshold in EEGLAB stats more lenient, we get something more similar to this for the 1st independent:

From this, it would seem that:

• the 1st independent using EEGLAB stats = alcohol vs no alcohol
• the 1st independent using Fieldtrip stats = go vs no go
• the inverse is true for the 2nd independent.

So, I conclude two things from this:

• In a 2x2 design, if you calculate the effect of only 1 independent, the actual factor used is swapped between the EEGLAB and Fieldtrip statistics option - which seems undesirable, if true.
• The labeling of the Fieldtrip stats corresponds more to intuition: GoNoGo shows you the effect of this factor, not the effect of the other factor "when you collapse across the one mentioned in the title". Such a way of naming things makes sense in the marginal effects plot:
  
  but can lead to critically erroneous conclusions when only considering the interaction plot:
  

For completeness, if ignoring the alcohol factor and making a 1x2 design, the same parameters give the figures below for EEGLAB and Fieldtrip stats, respectively. The effect labeled "GoNoGo" here seems identical to the one labeled "Alcohol" in the 2x2 analysis for EEGLAB stats:
EEGLAB:

Fieldtrip:

[arnodelorme]

I think the real way to compare is to use the same stats. This is for the first condition EEGLAB on the left and Fieldtrip on the right)

This is for the second condition. EEGLAB on the left and Fieldtrip on the right)

Bonferoni is much more aggressive than cluster correction, which could explain the difference in your case.

[arnodelorme]

I am closing this report, but let me know if you have more questions.

[KrisBaetens]

Dear Arno, Thank you for looking into this. Using the same stats certainly makes the presentation clearer, but the issue remains the same. As illustrated below, using .05 param / analytic, the results for compute 1st independent variable using EEGLAB stats correspond to compute 2nd independent variable using Fieldtrip and vice versa. (Moreover, using EEGLAB stats, what is labelend "GoNoGo" is actually the sites of significant effects of the "Alcohol" factor and vice versa.) Compute 1st independent variable, EEGLAB stats: [cid:07c1a943-651e-4c94-b531-7873c7ce879a] Compute 1st independent variable, Fieldtrip stats: [cid:f82b96b1-e916-4a1c-8586-faee21fcafa4] Compute 2nd independent variable, EEGLAB stats: [cid:1ff728e2-623a-4dca-83f6-67aee25764a5] Compute 2nd independent variable, Fieldtrip stats: [cid:e8857c91-5dbe-4ae4-82fb-fc12039c32df] Kind regards [cid:79284dbb-4f2b-4b80-91ed-e90b2cb33029]<http://www.vub.ac.be/> Prof. Dr. Kris Baetens Faculty of Psychology and Educational Sciences E ***@***.*** T +32 (0)2 629 23 31 Room B.3.01B - Pleinlaan 2 - 1050 Brussel www.vub.ac.be<http://www.vub.ac.be/> HET DENKEN ONDERWERPT ZICH NIET #WeAreVUB Schrijf je in voor onze nieuwsbrief <http://communicate.vub.ac.be/subscription-center-nl> - Sign up for our newsletter<http://communicate.vub.ac.be/subscription-center-en> - disclaimer<http://huisstijl.vub.ac.be/styleguide/disclaimer>

…

________________________________ Van: Arnaud Delorme ***@***.***> Verzonden: woensdag 12 februari 2025 17:31 Aan: sccn/eeglab ***@***.***> CC: Kris BAETENS ***@***.***>; Comment ***@***.***> Onderwerp: Re: [sccn/eeglab] Issue with study stats (Issue #850) I am closing this report, but let me know if you have more questions. — Reply to this email directly, view it on GitHub<#850 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/BPNPYJ44ITBXLBRF6ZK5CGD2PNZP5AVCNFSM6AAAAABW6F6WKSVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDMNJUGI2DSNZSGU>. You are receiving this because you commented.Message ID: ***@***.***>

[arnodelorme]

Would you mind reposting with the images (they do not show up). Are you using the latest version of EEGLAB? There was a problem that was fixed a couple of months ago. Would you mind trying with the version below.

https://sccn.ucsd.edu/eeglab/currentversion/eeglab2025.0.0rc.zip

[KrisBaetens]

Dear Arnaud, I reran in v2025.0.0. The outcome is the same. (I attached the figures in word - I have little control over email settings in this institutional mail). Included are: - the 4 individual outputs of the 2x2 design, each time plotting only the effect of one of the independent variables - for completeness, below, using a simpler 1x2 design, the results are virtually identical for the selected factor, but only the Fieldtrip ones align with the output of the 2x2 Thanks again for your wonderful work! Kind regards

[arnodelorme]

Unfortunately, I cannot see the word doc on Github. Can you send it to my personal email at arnodelorme at gmail dot com. Also, is there a way to access the data to reproduce the issue?