jsPsych versions of Classic Experiments

[pjkohler]

Hi Everyone,

it would be nice to have jsPsych versions of some classic experiments (Stroop, Visual Search, Ebbinghaus etc.) that could be used for teaching laboratory courses and things like that. I know that many of these are available in PsychoJs-versions on Pavlovia, but is anything available based on JsPsych (which is my preference)? And if someone made one or more of these, what would be a good way of sharing with the community?

Best,

Peter

[jodeleeuw]

Hi Peter,

I'd love to have this as well. Several years ago I started to create something like this, which I called factorsdb. It was a Jekyll-powered website with jsPsych experiments aimed at demonstrations for the classroom. I didn't get too far because other priorities with jsPsych development and PhD work pushed this aside.

I think there are a few ways that something like this could be shared.

• We could integrate it into the jsPsych docs, especially if we solved Link to live examples from the docs. #164 and hosted live examples somehow. We could add lots of tutorials, or simply a new section of demo experiments that contained both code and live demo.
• We could have a separate repo in the jspsych organizations that was structured around demo experiments. We could use GitHub pages to deploy live versions of these experiments, with a very simple indexing page.
• We could just expand the /examples folder in the jsPsych repository to include more full experiments. There are a couple longer experiments in that folder now. We could even add subfolders to better differentiate the many kinds of examples that now exist.

I'm open to any of these, especially if you and/or others are going to take the lead on setting things up 😁.

[batshell]

I would love to, I can even create some other examples as well.

[pjkohler]

Hmm ... - let me think about that. For me it would be crucial that whatever option we go with has a mechanism for actually collecting data using the demo experiments. I would want students to be able to run themselves and analyze the data offline.

[pjkohler]

of course, this could be done by simply grabbing the demo experiment from the examples folder and putting it on cognition.run, right?

[jodeleeuw]

If students are just capturing their own data then we can use something like jsPsych.data.get().localSave() at the end of demo experiments to allow downloading of that run's data.

If you want to allow data aggregation across a group of students, then we're in the territory of needing a server that has data storage and I think it gets a lot more complicated. Definitely a worthwhile thing, but a lot more resource intensive.

[jodeleeuw]

Yes, cognition.run would be a good host for self-deploying an example experiment in a teaching context. We could put together a tutorial that explains how to do that with the examples.

[pjkohler]

I think for it would make most sense to make it separate from examples. I think the documentation for plugins would note which demos use the plugin, that would be neat.

[pjkohler]

I think I would be willing to organize the effort, and also code some of the experiments. Should we do some kind of announcement to spread the word on this, see if other people would be interested in contributing experiments?

[jodeleeuw]

🥳

I think what might be most effective is to produce a minimal demo of what this will look like, even with just a single experiment, and then announce that with an invitation to participate. People will have a clearer idea of what format they should be aiming to contribute in. And we can publish clear contribution instructions.

[pjkohler]

I can work on a few experiments on my end. But what about organization?

Would we want a subfolder called demos in the main jsPsych repo, for example?

[jodeleeuw]

I'm leaning in the direction of having this be its own repo. What I am worried about is that some experiments may break after we update jsPsych. If this project is its own repo, then each experiment can have its own copy of the right version of jsPsych, and a readme file that describes which version it uses. Of course we can update these as new versions are released, but I don't want to create a situation where we have to update potentially dozens of experiments when we publish a new release.

Maybe my repo for YouTube tutorials is a good model of organization?

[pjkohler]

Okay. Do we want to set up the demos repo under the jspsych organization? It seems to me that we would want the demos to all "point to" the same jspsych folder, so would we want a jspsych subfolder under the demos repo? Perhaps as a submodule of the main jspsych repo?

[jodeleeuw]

I think it might actually be better to have the demos have their own copy of jsPsych. It ensures that each demo is self-contained. We could have one folder for each release I suppose, but this makes it a little harder to grab a chunk of the repo and launch it on one's own. We can delete all the plugins and examples that aren't needed from the release, so that it is very lightweight to copy the library multiple times.

I've created a repo and invited you to have maintain access to it.

[pjkohler]

sorry, @jodeleeuw, I realize now that under the youtube tutorials repo you have separate jspsych folders for each experiment. That seems kinda ... "wasteful" ... but I guess it is the only way to allow separate versions of jspsych to be used for each demo.

[becky-gilbert]

Great idea @pjkohler! I have some classic experiments/tasks that I'd be happy to contribute.

I also wanted to mention that the experiment factory can be a good resource for jsPsych versions of classic experiments, in case you're not aware of it. Probably not exactly what you're looking for, since these experiments use some custom plugins and config stuff that is specific to their system. But their code can sometimes be a useful starting point.

[pjkohler]

@becky-gilbert: thanks for the heads up re. experiment factory, definitively a potential source of inspiration / code.

[pjkohler]

Would people that want to contribute an experiment do pull requests to the experiment-demos repo?

[jodeleeuw]

I think that would make sense

[steffejr]

Great idea. I will be contributing whatever I do. All of my jsPsych code is going here:
https://github.com/NCMlab/WebbasedCognitiveTasks

As of right now I have the Stroop Word, Color, ColorWord tasks all complete and in good shape. I just want to make some of the parameters modifiable via a config file. I like how millisecond lays out their instructions about what the tasks do and what parameters are modifiable. I am going to try to work towards that.

I have a list of my priorities which I will work on next. For instance, next will be arrow flankers. Then a verbal delayed match to sample.

Jason

[pjkohler]

Hi @steffejr

Very nice to e-meet you. I see that we are in the same province (I am at York University).

Your contribution would be fantastic. I am not that familiar with millisecond, but we are very open to adopting standards of presentation across the demos and I am certainly happy to help contributors adapt their experiments to these standards.

Do you envision individual demos for every Stroop task, or do you have a single demo where the parameters can be changed to run different versions of the task?

I would like to include Shepard's mental rotation task, and as previously mentioned, an Ebbinghaus size judgment task. I will work on those.

Best,

Peter

[steffejr]

Hello Peter,
Good to meet you too and that is funny that out of the whole world we are not far apart. And neither is Josh.
Millisecond has a huge library of tasks and they seem to have a standard of documentation.
What I am going to do is to include a description file for each task. Here is an outline for Stroop: https://github.com/NCMlab/WebbasedCognitiveTasks/blob/main/Stroop/StroopDescription.md

I am also going to use a config file for each task where a user can edit a lot of the parameters of the task. Here is the start of such a file:
https://github.com/NCMlab/WebbasedCognitiveTasks/blob/main/Stroop/StroopSetup.js

As for the Stroop, I am using three different files. One for color, one for word and one for color/word. I am using a single config file that will be imported to each task. This way only one config file needs to be modified.

later,
Jason

[pjkohler]

Hi Jason,

Looks reasonable. As per our previous discussion, your demo folder should include the necessary jspsych files. Feel free to do a pull request to the experiment-demos repository, when you are ready.

Best,

Peter

[kurokida]

@pjkohler , Thank you for letting me know this project.
I think this is a great plan!
In fact, in Japan, we have published classic psychological experiments made using jsPsych for the educational purpose. (Sorry for Japanese only.)
The owner of this project is @kohske
For this project, I made a stroop and a mirror drawing task programs.
The stroop program seems to have been created by others, but the mirror drawing program may be rare. I can provide the program.

[pjkohler]

The mirror drawing program would be a fantastic addition.

[kohske]

This project sounds great. I'm also happy to contribute :)

[steffejr]

Does anyone have any good ideas for coordination? This way we don't have two people developing the same tasks? I know I am starting with Stroop because it is a great way for me to learn the details of jsPsych.

[pjkohler]

Good point, @steffejr. Could I create projects under the repo and assign them to specific people? @jodeleeuw do you have other ideas?

[jodeleeuw]

How about Issues in the repo? If someone wants to build an experiment they can open an issue, and close it when it is done. That will also make it clear when an experiment has been started but abandoned before completion. Perhaps use the repo's README file to specify guidelines for contributing, including this process?

[steffejr]

Under issues we can use a label such as DemoTaskDevelopment and then add an assignee, which would be ourselves. I have never used these features before but they look like they would work.

[pjkohler]

Me neither ... what you are proposing sounds great.

[pjkohler]

yeah, I just assigned @becky-gilbert and myself to our respective issues, and assigned the DemoTaskDevelopment label to both. Seems like an ideal system. I will try to write some documentation on the procedure in the not so distant future.

[pjkohler]

Sounds good.

[becky-gilbert]

I wonder if there's a straightforward way for this repo and/or the individual tasks to be cited? Maybe there could be something on the readme about this (and a note to remember to cite the jsPsych paper as well). Or at the very least, perhaps a contributors.md file in the root directory?

[jodeleeuw]

I think this is a really important thing to think about. I think a readme template that is copied for each experiment that includes citation info would be a good step.

[joshkenney]

@pjkohler I have developed a very robust version of the Ebbinghaus illusion and many other tasks. Over the last year, I've refactored 15 tasks into jsPsych from various software (ePrime, NBS, PsychoPy, MATLAB) and have begun to develop a framework around jsPsych which has proven robust across the 15 tasks.

The framework provides an intake screen prior to task presentation, database and experiment configuration files, all using common directory structure and namespace.

It would be fun to share what I've been working on with this group. It seems like we are all working towards a common goal of aggregating a task library. I really enjoy using jsPsych and want to help increase its adoption.

I'm happy to walk you around our ebbinghausIllusion repo and add you as a collaborator if this is something you're interested in.

Lemme know - Josh Kenney

[steffejr]

Hello Josh, This sounds like it would be great. Over the years I have refactored tasks from EPrime to MatLab to psychopy and now to jspsych. I made a nice wrapper and GUI for all of my psychopy tasks too. My plan is to use JATOS with the jspsych tasks. I would love to see what you have also. This is exciting, Jason

…

On Sat, Mar 6, 2021 at 10:04 PM Josh Kenney ***@***.***> wrote: @pjkohler <https://github.com/pjkohler> I have developed a very robust version of the Ebbinghaus illusion and many other tasks. Over the last year, I've refactored 15 tasks into jsPsych from various software (ePrime, NBS, PsychoPy, MATLAB) and have begun to develop a framework around jsPsych which has proven robust across the 15 tasks. The framework provides an intake screen prior to task presentation, database and experiment configuration files, all using common directory structure and namespace. It would be fun to share what I've been working on with this group. It seems like we are all working towards a common goal of aggregating a task library. I really enjoy using jsPsych and want to help increase its adoption. I'm happy to walk you around our ebbinghausIllusion repo and add you as a collaborator if this is something you're interested in. Lemme know - Josh Kenney — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#1575 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/AAIWBI4ZBDLPEL5R6RYHPZ3TCLULTANCNFSM4YHROAQA> .

[belieflab]

@steffejr thanks for the reply!

Would be great to combine our efforts like @pjkohler suggested below. Perhaps we can get together soon and share what we have developed independently. When is a good time to meet?

[joshkenney]

oops, replied with the lab github - either one works

[pjkohler]

Great to hear from you @joshkenney. Both @steffejr and yourself mentioned have a common structure (for experiment configuration and data storage) across many experiments. Do you think we could find a structure that both of you would agree to? If so, we could make it the standard structure for our experiment demos.

The way I see it, we have a choice of either adopting a standard structure that has a certain complexity, or enforce / encourage submissions to be a simple as possible. I would worry about a structure where some experiments have one structure and others have another - would be confusing to work with.

I don't think the structure should be mandatory for new submissions, but we would document it and encourage people to use it. In some cases we could work with authors on getting new submissions into that structure - I am certainly happy to do that.

As for collaborating on other repositories, that would certainly be educational, but I think I would prefer that people follow the procedure suggested above:
(1) announce a planned addition to the experiment library by raising an issue.
(2) make a pull request when the new addition is ready.

Very Exciting, indeed.

/Peter

[joshkenney]

thanks @pjkohler. Yes, I agree that a common structure is needed. I had spoke with @jodeleeuw and @becky-gilbert last fall about the possibility of integrating such guidelines into the jsPsych documentation. I would argue for adapting a structure where most of the complexity is abstracted and the users are encouraged to navigate a standard directory structure. I've found that splitting the experiment up into its component parts (timeline, global variables, function definitions, and a callable main) has made it easier to program over time and reduces the likelihood of runtime errors.

There were a lot of links posted in this thread, can you link to the repository you mentioned above?

[pjkohler]

https://github.com/jspsych/experiment-demos

I am writing the Readme as we speak. It seems you guys have all the experiments done already, so I gotta contribute something! 😄

Speaking of, does anyone have a version of the Shepard & Metzler mental rotation study?

[joshkenney]

@pjkohler thanks for the link!

I've cloned the repo and added the ebbinghaus repo as a submodule within experiment demos. I'm all set to push. Do I need to be added as a collaborator?

I haven't done a pull request in a while; I just created an issue for ebbinghausIllusion. Forgive me if I might need some handholding, here.

[pjkohler]

GREAT. Happy to handhold. I replied to your issue.

[steffejr]

Hello all, I just found this: https://github.com/expfactory-experiments
It is from Russ Poldrack's at Standford.

[pjkohler]

Yeah, I knew about this. Not so geared towards getting people into jsPsych, though.