Requesting feedback on a new online experiment platform

[javidalpe]

TL;DR I just released https://www.cognition.run/ a free platform to help scientists to run online experiments using jsPsych. I need your feedback!

---

First, I would like to thank @jodeleeuw for this amazing library. As an experienced engineer not related to science, I have to say that this library is first-class: easy to understand, easy to implement, easy to integrate, and very well maintained. If I had to program a cognitive experiment, I'd choose this library with no doubts!

---

I'm presenting cognition.run, a platform to run online cognitive experiments using jsPsych.

How does it work?

1. Create a task: copy and paste your code, or use our online editor, to create your task using jsPsych
2. Every task gets a link to run the experiment.
3. Share the link with your participants.
4. Cognition.run collects the experiment's data for you.
5. Download the data in CSV or JSON format.

Why?

My wife is a neuroscientist researcher in California. She used to create/design her experiments using Psychtoolbox (Matlab), but now she has to move her experiments to online due to the COVID-19 situation. She was looking for good alternatives, so she asked me for help, and then I found jsPsych. I fall in love with this library, but I couldn't find an easy solution to connect the front with a backend, so I decided to start this simple PHP+Bootstrap theme+Mysql project to help others.

[vijaymarupudi]

I haven't tried the internals, but I think that homepage looks gorgeous!

[jodeleeuw]

@javidalpe this is amazing!

I'm honestly just blown away at how smooth and easy this is to use. The task preview and data preview are fantastic. The automatic plugin detection is a neat trick. (Note that in some very advanced cases the type parameter could be a function or a timeline variable that's updated at runtime, so for completeness you may want to allow users to manually include a plugin. This is also true if someone has a custom plugin that's not part of the official build.)

One thing that would be really nice would be the ability to sync with a git repo so that someone could take a task that was developed in another context and bring it in quickly. That might be really tough to configure given the way you've set things up. I'd at least like to be able to get my task code out of your IDE so I can do things like publish a public version of the code on the Open Science Framework.

I've been thinking about developing a freely hosted platform like this for sometime now and I'm thrilled that you've made that obsolete! A million thanks. This is a true game changer! I'm worried that this might get more traffic than you are anticipating and the hosting costs might jump significantly. Do you have a plan in place for that?

[javidalpe]

Thank you so much @jodeleeuw ! Happy you liked it! And congrats again. This is thanks to the architecture of the library :)

Regarding Git sync, I was asking my self: how can I improve the UX of the deployment for someone who is working locally and doesn't necessarily know GIT. I was unable to find a good solution better than copy&paste the javascript and upload the stimuli. Then, I realized that I could integrate an online editor and let the user code online. By doing this, I could remove the deploy phase and improve the developer experience -live-reloading, data preview, ...- at the same time. Yeah, I'll definitely add the "Export" feature.

Everything is hosted in AWS, which is pretty cheap. We have a plan with different scenarios. It will depend on the feedback and the noise we get from the project!

[hauselin]

Incredible website/platform, @javidalpe! I've been doing something similar, mostly because I'm trying to learn how to set up a noSQL/mongo database and use jsPsych. I just tried a few of my jsPsych studies on it and it works really well.

Thanks also to @jodeleeuw for this amazing library (and great documentation)—it's so much more intuitive than any other library/package I've used (e.g., PsychoPy). Clearly a lot of thought has gone into making this library.

Thanks again to both of you for your incredible work!

[kupiqu]

I think this is a great initiative and a very nice addition to jsPsych.

To give a bit of feedback, first, I'd like to second what others have said above and this in particular:

One thing that would be really nice would be the ability to sync with a git repo so that someone could take a task that was developed in another context and bring it in quickly.

And second (and somewhat related to this), I would also encourage adding the possibility of uploading a flavored jsPsych library as part of the experiment. Sometimes an experiment relies on fixes that are only present in master, or even in not yet accepted pull requests, so having such option would add flexibility if/when needed.

[javidalpe]

Thank you! I love your feedback ❤️ !

I'm thinking about how to do it 🤔 :
A) Do nothing. The user uses "External libraries" to upload his custom version. Uploaded external libraries execute after jspsych official lib, so it should override jspsych global variable. cc @jodeleeuw
B) On "JsPsych library version:" selector, add a "Use my uploaded version". Don't load official jspsych lib. Display an error if no jspsych global variable is found during runtime.

What do you think?

[Stajduhar-A]

First of all, I have to say this is absolutely incredible! Thank you so much for creating such an easy-to-use site! I have to say I like option B from above because, while there are some edited plugins I would love to use, I haven't edited them all and I'd like the option for the default to be stock plugins unless otherwise specified (i.e uploaded by me). That kind of flexibility would be truly amazing. Thank you again for the time and attention you've put into this.

[becky-gilbert]

I have a bit of feedback but mainly just want to echo what everyone else has already said - this is AMAZING! The whole code-preview-data IDE is fantastic. I love it.

I'd like to second this comment from Josh:

so for completeness you may want to allow users to manually include a plugin. This is also true if someone has a custom plugin that's not part of the official build.

I frequently use custom and modified plugins, so this would be important for me. Although it looks like that might be possible from inside the Advanced Editor, where it says "External libraries", as it looks like this allows you to add any .js or .css file. Is that right? (Sorry that I haven't had a chance to try this yet).

Also, I didn't see a T&Cs section anywhere - did I miss it? I wonder if you should add something saying that you're not liable for misuse of the platform, e.g. if someone runs a study without ethics approval. Admittedly I know nothing about this sort of legal stuff, so might not be an issue.

Regarding the hosting costs and time spent on server admin, I wonder if people can help support this platform somehow. Maybe you could add a donation area/link onto the site? Is there anything else that the jsPsych community could do to contribute?

[javidalpe]

Thank you very much @becky-gilbert.

Maybe you can add your custom plugin at the beginning of the main script, or use the 'External libraries' as you mentioned. I'd love to hear from you once you try the "External libraries" feature. I want to do it as simple as possible w/o losing flexibility.

Good point about T&C! It is something we should work on in the next days.

Thank you again for the offer. The hosting cost is more than affordable for now, so it is not a priority. Our top priority right now is to build trust, so we were wondering if you could send us a "testimonial" that we can add to the website. I’m not looking for a novel or anything. Just a few sentences describing your experience with cognition.run.

[becky-gilbert]

Great, thanks!

In that case, one very minor suggestion would be to change the label of the file import area in the advanced editor from 'External libraries' to something like 'External JS/CSS files' (I realise you have some default text in the file upload box that says something like 'Upload JS or CSS files', but that text is only partially visible for me, and I can't see how to resize the left-hand column...).

I hope to have more time to use cognition.run soon, and then I'll definitely send a testimonial! 😃

[tobiasgerstenberg]

this looks excellent! i'm leading the causality in cognition lab at stanford and we are currently using psiturk to run our experiments (hosted on an AWS server).

i'm considering switching to cognition.run + jsPsych. however, here are some features that are critical for me, and i was wondering if these could be implemented:

1. run HITs in small batches of 9 to avoid the additional mturk fees
2. easily exclude repeat participants (not only from the same study, but from other similar studies, too) -- in psiturk this is handled by using a single database for each project that then makes sure that no repeats appear within this project (which might contain several different experiments)
3. automatic assignment of participants to different conditions (i.e. i can run an experiment with 3 conditions and psiturk makes sure that i have an equal number of participants in each condition)
4. have a permanent experiment link that can be shared in a scientific paper (so that a reader can easily play around with the experiment; this version of the experiment shouldn't be collecting any data anymore) [note: this is currently not possible using the psiturk setup]
5. ideally: the ability to (easily) host experiments not only on mturk but on other services, too (such as prolific) [also not possible with psiturk]

[javidalpe]

Hey Tobias, we have news for you 👍 :

1. We consider this feature out of scope for cognition.run 👎 . We are a hosting platform for serving experiments. You can see it as "the Survey Monkey for cognitive experiments". A mTurk integration is not in our roadmap right now. You will need another service to manage mTurk like CloudResearch.
2. We are adding this feature to our roadmap as a low priority 👍 since this feature is already available in other services like CloudResearch or mTurk.
3. We will implement this feature in the near future 👍
4. We will implement this feature in the near future 👍 . It will be as simple as adding a URL param. Ex: https://ymdfgdcuo9.cognition.run?demo=true
5. This is already possible 👍 . Every experiment gets a secure URL ready to share in any platform.

[tobiasgerstenberg]

great, thanks for the update!

[javidalpe]

Hello Tobias 👋 ,

We have implemented 3) and 4) 🥳 . Now you can define conditions and avoid storing data (demo mode) using ?demo=true at the end of the experiment URL.

Let us know if you have more questions or concerns.

cc @jenniekim00

[jodeleeuw]

Awesome! Looks like condition information is available in the global variable CONDITION. Is that right? Are there docs we can point people to?

Can you enable control over the CONDITION variable in the source code view to enable easier testing?

[javidalpe]

Awesome! Looks like condition information is available in the global variable CONDITION. Is that right? Are there docs we can point people to?

Can you enable control over the CONDITION variable in the source code view to enable easier testing?

That's right @jodeleeuw 🌟 I forgot to mention 😅, so thank you for the clarification. We have no docs right now, but we are planning to add a doc page in the near future.

Sure, that's a great idea 💯 . I'll add a 'Set condition' in the preview tab.

[jenniekim00]

Hello, I have just discovered Cognition.run this past week and let me say THANK YOU!!! A god send, life-savior, you all are amazing!

I recently tested my code with cognition.run and I did experience a few issues, all that were solvable in one way or the other (technical and nontechnical). I'll share my initial experience in hopes it might help you guys as well!

Issues and Feedback:

1. I copied and pasted into cognition.run with the code I have been working on, which is currently around 900 lines of code. I had to zoom out my browser at 25% to see my whole code, task preview, and recorded data section. This made it difficult to code within the interface and I would just edit my code with another IDE (atom) I have been using, and re-upload the file or re-copy-and-paste. There is a delay in when the task preview loads, and the center of the screen small in comparison to the rest of the screen when I zoom out. This might just be a problem of handling larger loads of code.

2. When I run my code locally on my computer, at the end the csv file automatically downloads at the end of my experiment. On cogntion.run, the data is recorded but all the tasks are defined as 'unfinished' and doesn't say 'csv download' on the main feed even though the csv did save onto my local computer and the data is shown in the 'view' section. I can still download a csv file from the data shown in the 'view' section, but I am just confused why the main data section feed doesn't register the experiment as completed and gives no csv download file link/click (as shown in the youtube tutorial by jodeleeuw).

3. very minor but 'ctrl f' doesn't work inside the javascript code interface aha

4. Originally without the cognition.run, I had my javascript code within <script> tags in an html doc, and struggled to figure out how to insert script src links that were url based and initially gave me errors that 'jstat' was undefined. Still same issues when i uploaded an external copy of the js code from those links. For example, the script src links I'm referring to:

<script src="https://cdn.jsdelivr.net/npm/jstat@latest/dist/jstat.min.js"></script> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>

Luckily there is a solution, where I just added this code to the cognition.run javascript interface:
const script = document.createElement("script")
script.type = "text/javascript"
script.src = "jStat.js"
document.head.appendChild(script)
const script1 = document.createElement("script")
script1.type = "text/javascript"
script1.src = "jquery.min.js"
document.head.appendChild(script1
eval(script)
eval(script1)

5. Also more preference of UX, but it'd be nice to have a drop down menu not to just go to 'tasks' but also the subsections like 'source code' and/or 'go to/open shareable link'. Because I have been having issues with my screen being zoomed in/cut off and hard to see zoomed out, I have been testing my code through the shareable link instead of the task preview. And thus, it'd be nice to have an option to go to the shareable link without having to go out of the task coding interface.

6. For the Informed Consent Form section, it would be nice if there was a "Save" and or "Delete" button in the interface, and same with a 'Saved' feedback for the coding section, since there is a delay for me and I need to refresh with uncertainty if the code was actually updated before i check the shareable link.

7. On the topic of the Informed Consent Form, this is more irrelevant to cognition.run's functions and more so of my code. But I experienced a few issues with the interaction of one plugin when inserting an informed consent through cognition run. The NASA-TLX plugin, that the awesome vijaymarupudi manipulated conveniently into jspsych format, started popping up on the same screen (above) as the Informed consent, but this issue does not occur when I do not use the informed consent (inserting text into that section trhough cognition.run). I think it's more so how the NASA TLX plugin is structured. Luckily I can solve this issue by coding an informed consent page into my code, since I am not smart enough to figure out the technical fix within the NASA TLX plugin aha.

8. It would also be nice, as tobiasgerstenberg suggested, if there was a feature to automatically assign participants to different conditions. My current nontechnical method I'm going for is to create two 'tasks' with cognition.run for my between subjects conditions, and somehow assign them participant numbers outside my code, and have them use one of the shareable links depending on if they're P# is even or odd.

Thank you guys for all your awesomeness. Without jspsych, without cognition, run, without all the helpful collaborators on here... I would not be where I am with my thesis during this pandemic. Thank you.

[javidalpe]

Hey @amunn, I've been investigating this issue for two days and still don't understand what is happening here. I can reproduce your error but I don't get it.

I have noticed that it works fine if you disable cognition stimuli preloading (located at Advanced configuration). Maybe I have to figure out how other plugins interact with the library preloading feature, and why paths make the difference.

[amunn]

@javidalpe Thanks! That helps for now. Feel free to contact me if you need any other help with testing.

[GEJ1]

<script src="https://cdn.jsdelivr.net/npm/jstat@latest/dist/jstat.min.js"></script> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script>

Luckily there is a solution, where I just added this code to the cognition.run javascript interface:
const script = document.createElement("script")
script.type = "text/javascript"
script.src = "jStat.js"
document.head.appendChild(script)
const script1 = document.createElement("script")
script1.type = "text/javascript"
script1.src = "jquery.min.js"
document.head.appendChild(script1
eval(script)
eval(script1)

This is a very smart solution. Just two tips. 1) You don't need to call "eval" to get the code running on the browser. 2) The scripts tags will be added to the header and the execution will continue. This means that any code that depends on the external file should wait until the script is loaded or will fail with some short of "blablbla is not defined". If you need to wait for this external script to load before executing any custom script, I recommend you to listen to the onload event on the script tag. The code could be like:

var script = document.createElement('script');
script.onload = function () {
    //do stuff with the script, like jspsych.init()
};
script.src = "HTTP://utl_to_external_js";

document.head.appendChild(script);

That said, my recommendation is to download in your computer any external library (jquery, bootstrap, loadash, ...) and upload it to cognition.run, so the platform will host the files and serve them before any custom script 👍

Hello @javidalpe ,

First of all, congratulations for the awesome project I thinks this is the perfect complement for jsPsych 👍 .

I have a similar problem with jquery-ui.css in cognition.run. It works in my computer but it didn't on web. I downloaded the file and upload in the regular way but didn't work.

Trying to debbuging I saw that if I change the " rel="stylesheet" when I import the css file it doesn't work anymore:

works:
<link href="C:\Users\Gustavo\Desktop\jquery-ui.css" rel="stylesheet">

Not working:
<link href="C:\Users\Gustavo\Desktop\jquery-ui.css>

It is possible that cognition.run needs that rel parameter? In that case how can I added to JavaScript?

Thank you in advance!!

Best,

Gustavo.

[javidalpe]

Hola Gustavo! Thanks!

Yes, we are adding the rel parameter for stylesheets. Many things could be happening here, but I think the issue is related to the order you're loading the libraries. We load the libraries in the same order you upload them to the editor. The first you upload, the first we load.

If you want to check the code cognition.run is generating on your experiment, just right click on the website and then View Page Source. Also, check the browser console to find possible javascript errors.

Regards,
Javi

[GEJ1]

Hola Gustavo! Thanks!

Yes, we are adding the rel parameter for stylesheets. Many things could be happening here, but I think the issue is related to the order you're loading the libraries. We load the libraries in the same order you upload them to the editor. The first you upload, the first we load.

If you want to check the code cognition.run is generating on your experiment, just right click on the website and then View Page Source. Also, check the browser console to find possible javascript errors.

Regards,
Javi

The issue was related with the order of the libraries. Now is working!!

Thank you very much!

[rgriscom]

Wow, what a great tool! I know a number of linguists who work in somewhat remote areas and are interested in running experiments on mobile devices - this makes it easy.

[yuvalharr]

Thanks @javidalpe for a very awesome tool!

I have an issue not regarding the software itself which is great.
For some reason sometimes while on the 'tasks' page I try to press the 'source code' button but nothing happens and I'm not able to access the source code again. I just open a new task each time it happens.

Also, I tried looking for a way to contact you in the cognition.run website but couldn't seem to find any way to do so. So this will be great as well :)

Again, thank you very much for all your work!

[amunn]

Thanks @javidalpe this site is really fantastic. One request, if possible: would it be possible to allow the stimuli part of an experiment to have a folder structure? It seems right now that stimuli have to be a flat list of all the items. For experiments with lots of audio/images etc. this gets unwieldy quite quickly and also usually entails rejigging paths in the task since during development such files are likely in folders.

[javidalpe]

Hello @amunn.

This is a very good feature request. When I started building cognition.run, building a folder-based structure seemed "required" for me as a web developer. But, in order to improve the developer experience for non-expert developers, I decided to go with the flat approach.

Why?

As you can see at https://www.w3schools.com/html/html_filepaths.asp, there are many ways of referencing resources on HTML. While this is trivial for me, this is one of the most common sources of problems/errors/frustration when dealing with HTML for the first time. @jodeleeuw could probably agree with that.

How can I reduce the number of errors/support tickets/issues? - I thought.

So, given a flat structure, I can improve the developer experience by: 1) eliminating the mental model needed by the developer to understand the path system (what is a folder, which is the root folder, etc) and 2) redirecting any request to the flat directory and avoid 404 errors. So, any stimuli like:

<img src="picture.jpg">
<img src="images/picture.jpg">
<img src="/images/picture.jpg">
<img src="../picture.jpg">

Cognition.run will transform it to:

<img src="picture.jpg">
<img src="picture.jpg">
<img src="picture.jpg">
<img src="picture.jpg">

---

That said, I'm completely open to new ideas and approaches. I understand your needs and I would like to build a product that is equally useful for both experts and non-experts.

[amunn]

Thanks for your reply. Just so I understand things correctly: as long as each file name is unique, any paths in my task will be adjusted automatically to effectively remove the path? So I don't need to adjust the paths in the task itself as long as I put all the relevant files in to the stimuli folder. Crucially then, this does require unique filenames, since Fillers/image1.jpg and Experiment/image1.jpg can't exist as two separate files any more. Ironically then, this imposes a different mental model on those of use who are used to using paths... :)

[javidalpe]

You are totally right 👍 We are decreasing the developer experience of experts in order to improve the experience of n00b developers.

This is a strategic decision. Thousands of experiments are transitioning to online, with thousands of scientists deciding which platform to use. They are not experts. Many are learning javascript for the first time. Therefore, the opportunity to make something disruptive depends on those users. We are completely focused on n00bs right now, making the platform as smooth as possible.

But we hope that, even with this new mental model, the overall experience is good enough for you to use cognition. If you are use to deploy online experiments, you probably already know how many other things cognition do better for you 🚀

[amunn]

Ok great! Thanks. It's a small shift which is easy to adapt to, and the automatic adjustment of paths is very good to understand. But the platform is fantastic, and I totally understand your focus on new users.

[yuvalharr]

Really great website!
I have one issue though. I added an on_finish function to the jspsych.init method (presented below), so that a message will be displayed in the end of the experiment. For some reason it doesn't display. When I run it locally on my computer it works fine and the message is displayed, so I guess there is some problem in cognition.run.

jsPsych.init({
    timeline: [introduction_procedure,practice_procedure, trial_procedure],
    on_finish: function() {
      alert("The experiment is over. Thank you for your participation!");
    }
});

Do you know of anything that can explain this?

Thanks!

[javidalpe]

Yes @yuvalharr, is a software regression error. It was working but a change in the code to enable labjs broke it. It is my fault, I'm sorry 😭 Thanks for reporting!

(Now it is fixed)

[yuvalharr]

Great, thank you very much!!

[jwylie21]

Hello!

This is amazing! I actually heard about Cognition.run before hearing about jsPsych, so a million thanks to both of you for these resources! This is exactly what i need to get some cognitive research off the ground.

I have programmed an experiment that runs locally as expected, but I can't seem to get it to run in Cognition.run. The three issues that seem to pop up in developer view are screen shotted below. Any idea how I can debug this? Additionally, I can share more information about my script if needed! THANK YOU!

[choppr]

Hi @javidalpe

Thank you for creating this framework!
I do have a small issue with the correct jsPsych library being used. The getKeyboardResponse function being used locally looks like this:

module.getKeyboardResponse = function(parameters) {
    //parameters are: callback_function, valid_responses, rt_method, persist, audio_context, audio_context_start_time, allow_held_key?

    parameters.rt_method = (typeof parameters.rt_method === 'undefined') ? 'date' : parameters.rt_method;
    if (parameters.rt_method != 'date' && parameters.rt_method != 'performance' && parameters.rt_method != 'audio') {
      console.log('Invalid RT method specified in getKeyboardResponse. Defaulting to "date" method.');
      parameters.rt_method = 'date';
    }

    var start_time;
    if (parameters.rt_method == 'date') {
      start_time = (new Date()).getTime();
    } else if (parameters.rt_method == 'performance') {
      start_time = performance.now();
    } else if (parameters.rt_method == 'audio') {
      start_time = parameters.audio_context_start_time;
    }

However in my Cognition.run experiment, the function looks like this:

module.getKeyboardResponse = function(parameters) {
    //parameters are: callback_function, valid_responses, rt_method, persist, audio_context, audio_context_start_time, allow_held_key?

    parameters.rt_method = (typeof parameters.rt_method === 'undefined') ? 'performance' : parameters.rt_method;
    if (parameters.rt_method != 'performance' && parameters.rt_method != 'audio') {
      console.log('Invalid RT method specified in getKeyboardResponse. Defaulting to "performance" method.');
      parameters.rt_method = 'performance';
    }

    var start_time;
    if (parameters.rt_method == 'performance') {
      start_time = performance.now();
    } else if (parameters.rt_method == 'audio') {
      start_time = parameters.audio_context_start_time;
    }

Am I using the wrong library? I have currently set it to 6.1.0

Thanks

[jodeleeuw]

I think your local version of jspsych is older than 6.1.0. We removed the "date" option recently.

[choppr]

Thanks I updated my local library and uploaded the latest instructions plugins and it works

[jwylie21]

Hello!

I am hoping to use JsPsych + Cognition.run to run an experiment using my campus's participant pool (SONA systems). Pavlovia and some other systems have integration features with SONA to allow for this type of data collection, and I'm wondering if there is either a way that I can do that (integrate with SONA so that students receive credit for participation) with the program in its current form, or if that might be possible to include in later versions.

Please let me know! Thanks SO much!!!!!!!

[ontinglo]

@javidalpe Thank you so much for creating such wonderful platform! i enjoy it so much!! really appreciate your efforts!

I am testing my online experiment now and encounter some problems, probably due to the images preloading. In my experiments, participants are required to judge 109 faces' trustworthiness by using a 9-point scale. It's a very simple experiment and the codes are tested perfectly in my local desktop computer, but when I run the codes on Cognition, I found there are abnormal time lags between stimuli.

I don't know how I could tackle this problem as the codes are ok in my local desktop computers.

Here is the link of my experiment
https://rgfaef1ujb.cognition.run

Could you please help me?
thanks a lot!

[ontinglo]

hi all, i think i've figured it out.

if we ask for image preloading in the jsPsych.init, then we should unclick the preloading choice provided by Cognition, or vice versa.
if we do it both in jsPsych.init and in the configuration of Cognition, it will be problematic.

thanks a lot

[javidalpe]

Hello Ontinglo. Thank you for reporting this. Thanks to your case I have found and fixed an issue regarding cookies and the database. While preloading the 109 faces, each request was generating a query to the database, refusing new requests at some point due to capacity limits.

And finally, to avoid the double preloading configuration problem, I have decided to do not override the preloading configuration provided by the user on code. If the user set preloading properties (preload_audio, preload_images, preload_video), the cognition preloading configuration will be ignored.

[amunn]

@javidalpe Thanks for this update. It's a very good solution to the preloading issue.

[rosemintz]

I'm having a similar issue with image lag. My experiment has audio and image preloading, and right now I'm just trying to run a toy version with about ten files in each. The audio is perfect but the images lag significantly in Safari (in Firefox, both are OK). Any idea what could cause this? I've tried all different combinations of my preload/Cognition's preload, and have not had an issue with Safari when the experiment was hosted elsewhere.

This is a super platform and I really appreciate everyone's efforts!

[LisaH1]

@javidalpe thank you so much for providing cognition.run, and thanks @jodeleeuw for the jsPsych library.

I just published an experiment for my master thesis and unfortunately, several people contacted me that the experiment does not load. They reach the consent form but the preloading of images and videos does not start. Instead, they receive an error message "the experiment failed to load" (reopening and/or changing browser does not help).
Since the code works for others (including me) and was tried on safari, edge, and chrome I cannot see the problem there. However, I also don't know what else could be wrong. Is it possible that some computers cause problems?

If you have any idea how to fix that I would highly appreciate your help.

[LisaH1]

Here is the link: https://7zghfo7z21.cognition.run

[becky-gilbert]

Hi @LisaH1, thanks for the info and for sharing your link. I just tried your link while throttling my connection to 5mbps, which is super slow for a non-mobile-data connection, and the media files took 5 minutes to load. So that seems like a good time limit.

And although your files load quickly for me on my normal internet connection, I noticed that there are quite a few image and movie files that together make up a considerable amount of data. So not only does that increase the chances of initial loading problems, but also problems with the files not remaining in the browser's cache. I'm not saying you will definitely run into that problem, but just in case you do: there's an explanation and solution on the "Preloading in batches" section of the Media Preloading documentation page.

Another thing I noticed in your task code is that your 'preload' trial comes before the 'welcome' message, which contains the note about needing to have a stable internet connection. So you could switch the order of those trials, and maybe warn people about that preloading will happen next (e.g. "The image and video files will be loaded when you click the next button. Please be patient while these files load and do not refresh the page."). You you can also set a custom error message with the preload plugin, which might be useful if you're worried that people will run into loading problems (e.g. "The files failed to load. Please try refreshing the page, or contact the experimenter: <your contact details>.").

[LisaH1]

Thanks for trying, I'm glad it still showed content after 5 min loading time, so it seems like adding the preload variable to my code has some effect at least. However, when I try to move the preload variable after the welcome message it still does the preloading before presenting the welcome page - I guess that's due to the design of cognition? Also, I cannot add a message and/or error_message to the preload plugin; it always just shows the loading bar.
The preloading in batches option is great but unfortunately not suitable for my experiment. I need to randomly pull videos from the whole set in the very beginning.

Thanks so much for your time @becky-gilbert ! I hope the 5 min max_load_time fixes the issues.

[javidalpe]

Hello Lisa and Becky. All this seems related the way cognition adds the preloading plugin under the hood. Please, checkout Task "Advanced configuration" to disable JsPsych Media Preloading.

---

Cognition JsPsych Media Preloading:
While this seems ok for many users, for some other could be a source of issues really hard to debug. For that reason I have decided to remove the "JsPsych Media Preloading:" feature from cognition. It will continue working for existing tasks, but you won't be able to add this feature to new tasks. I hope this makes things easier to understand and debug.

[becky-gilbert]

Thanks @javidalpe, that makes sense and is very helpful!

[hauselin]

@javidalpe, it seems like when I download the results as a zip file, the individual csv files in the zipped folder don't contain the extra columns with browser info (e.g., recorded_at, ip, user_agent... platform_version). But when I download the results as csv files, I do get those columns. Is this behavior intended? Thanks!

[hauselin]

It seems like when I download the results as a zip file, the individual csv files in the zipped folder don't contain the extra columns with browser info (e.g., recorded_at, ip, user_agent... platform_version). But when I download the results as csv files, I do get those columns. Is this behavior intended? Thanks!

Hi @javidalpe, could you check out this issue too? Thanks!

[jwylie21]

hello @javidalpe - there seems to be an issue for me accessing the "configuration" portion of my tasks. I have people taking my experiments (even though I am no longer sending out links), so I would like to disable them. Is there a way around this?

[csducate]

@javidalpe I'm having the same problem--it has actually been an issue on and off all summer. I've submitted a few bug reports but not heard back. Any idea what's going on? It seems to only be a problem with my big, "real" studies (i.e., my small sandbox test studies I can access the configuration page just fine). I get a 504 error.

[javidalpe]

For those who have had problems downloading data or have experienced stability problems: I'm happy to announce a revamp of the Download Data feature. Some highlights:

• There is no limit in the number of runs anymore.
• No limit on the total size of the data.
• CSV and Zip folder available.
• Files include IP and UserAgent info again.
• You can filter by all runs, only finished runs or only unfinished runs.

If you are curious about root of the issue, there was a bottleneck in the server. The server was downloading all CSV in memory and running the merge/zip operations. With large experiments the server was unable to respond in a proper time, and at the same time was blocking other requests. Now the download and the merge/zip operation happen entirely in the browser. This require some extra CPU from the user but greatly frees up the server.

[amunn]

Hi @javidalpe thanks for the update. Unfortunately the new version doesn't work quite the same as the old version. Duplicate columns in the data are now collapsed in a weird way. We had a condition column which was part of our experiment, but were also using the CONDITION variable to run different versions. Before the update, both columns were downloaded, i.e., we had a condition column with the cognition.run value, and a condition column with our values. With the new system we the condition column with our values where the cognition.run values would have been, and a condition column with no values where our condition column would have been. So effectively we lost the version numbers from 3 experiments and needed to manually recreate them. Global namespaces suck. :)

[amunn]

And yes, we have fixed things in our experiments so that there is no name clash :) but it was convenient that the previous version didn't actually mind that there were duplicate columns. Perhaps this was just an accident.

[javidalpe]

Hello Amunn. Thanks for reporting this issue. I have updated the merge algorithm to avoid this issue. While this shouldn't be a problem, just keep in mind that given two columns with the same name, the order of columns will determine the how the merge is done.

Please, let me know if you find any other issue downloading data :)

[amunn]

@javidalpe Another change that seems to have happened: jsPsych real valued columns (e.g. rt) are being downloaded now as integers instead.

[DanielBruijn]

@javidalpe Cognition.run is a great tool, and i've enjoyed using it so far. I do however have one particular frustration, namely that sometimes your code is not properly saved.
For example, when i finish working in the evening on my code, i close my laptop and assume the autosave function saves it. Then i start back up the next morning and see that the most recent changes have not been changed.
That has happened a few times now, so i decided to copy everything into a temporary text file as a backup. I'm sorry, i can't indicate under which conditions this seems to happen, but it might help you, me, and other users out if you know about it!

[javidalpe]

Hello Daniel! Sorry about that. I have added a "Saving...." spinner to the code editor while the code is sync with the server. I hope this helps while I try to figure out what was the main issue.

[DanielBruijn]

Oh, that is a great option, then you have some confirmation that it is saved. Just hypothesizing here, but a form of storing backups might be a good idea, that you can manually save a few versions of your code, just in case.
Good luck!

[LisaH1]

Hey @javidalpe!
I was trying to close my experiment for several days and cannot access the "configuration" option. People are still participating even though I cannot consider their data anymore. It would be great if I can switch it off, without deleting my code. In a simpler experiment, I can easily access the "configuration" and see the "switch off" option.
Thanks for your help!

[javidalpe]

Hi Lisa. We don't have any unexpected exception reported. May I know the link to the experiment or the url for the experiment configuration?

[LisaH1]

The link to the experiment is: https://7zghfo7z21.cognition.run

[python2020-maker]

Hi,

I pasted my javascript but doest work: the task is not ready yet.

My script:

<title>WM Digit Span</title> <script src="jspsych-6.0.4/jspsych.js"></script> <script src="jspsych-6.0.4/plugins/jspsych-html-keyboard-response.js"></script> <script src="jspsych-6.0.4/plugins/jspsych-digit-span-recall.js"></script> <script src="jspsych-6.0.4/plugins/jspsych-survey-text.js"></script> <script src="jspsych-6.0.4/plugins/jspsych-instructions.js"></script> <script src="jspsych-6.0.4/plugins/jspsych-fullscreen.js"></script> <script src="https://ajax.googleapis.com/ajax/libs/jquery/1.11.1/jquery.min.js"></script> <script src="https://cdn.jsdelivr.net/npm/jstat@latest/dist/jstat.min.js"></script> <script>

/*
This is a web-based digit span working memory test.
It is modelled after the forward digit span test described in Woods et al (2011) [https://www.ncbi.nlm.nih.gov/pmc/articles/PMC2978794/].
However, users can easily customize this test for their own purposes.
Easily customizable variables have been listed below. For further changes to the test, knowledge of JavaScipt may be required.
For smooth functioning of the test, make sure all the associated github files within the repository have been downloaded (especially the folder named 'jspsych-6.0.4').
Results from this test will be automatically downloaded into the downloads folder of your desktop.
For further details, please refer to the README.
*/

//----- CUSTOMIZABLE VARIABLES -----------------------------------------

nTrials = 14 // number of trials in the test
minSetSize = 3 // starting digit length
stimuli_duration = 1000 // number of miliseconds to display each digit
recall_duration = null // number of miliseconds to allow recall. If null, there is no time limit.
file_name = null // file name for data file. if null, a default name consisting of the participant ID and a unique number is chosen.
local = true // save the data file locally.
// If this test is being run online (e.g., on MTurk), true will cause the file to be downloaded to the participant's computer.
// If this test is on a server, and you wish to save the data file to that server, change this to false.
// If changed to false, ensure that the php file (its in the directory!) and the empty "data" folder has also been appropriately uploaded to the server.
// Incase of problems, feel free to contact me :)

//----------------------------------------------------------------------

possibleNumbers = ["1", "2", "3", "4", "5", "6", "7", "8", "9", "0"] //possible digits participants can get

var selection = jsPsych.randomization.sampleWithoutReplacement(possibleNumbers, minSetSize) //chooses random digits
var selection_id = -1

// keeps track of participant's scores:
var nError = 0
var highest_span_score = 0
var consec_error_score = 0

// first page. identifies participant for data storage
var p_details = {
type:"survey-text",
questions: [{prompt: "Enter participant number, name, or ID"}],
on_finish:function(){
partN = jsPsych.data.get().last(1).values()[0].partNum
partN = partN.replace(/['"]+/g,'')
// console.log(partN[0])
}
}

// instruction page
var instructions = {
type: 'instructions',
pages: function(){
pageOne = "

INSTRUCTIONS

This is the digit span task.

In this task, you will have to remember a sequence of numbers presented on the screen one after the other.
At the end of each trial, enter all the numbers into the onscreen number-pad in their correct order.

Example: if the sequence is '7 4 5 1', enter '7 4 5 1' in this exact order.

You will now be given practice trials to help you understand the task.
Press 'Next' to start the practice trials.

"
return [pageOne]
},
allow_backward: false,
button_label_next: "Next",
show_clickable_nav: true
}

var instructions_node = {
type: 'instructions',
pages: function(){
pageOne = "

You have completed the practice trials.

If you have further questions about the task, ask the researcher now.
If you are clear about the task, click 'Next' to proceed to the main trials.

"
return [pageOne]
},
allow_backward: false,
button_label_next: "Next",
show_clickable_nav: true,
on_finish: function(){
minSetSize = 3
selection = jsPsych.randomization.sampleWithoutReplacement(possibleNumbers, minSetSize)
}
}

var test_stimuli = {
type: 'html-keyboard-response',
stimulus: function(){
selection_id+=1
return '

'+selection[selection_id]+'

'
},
choices: jsPsych.NO_KEYS,
trial_duration: stimuli_duration,
on_finish: function(){
if (selection_id + 1 >=selection.length){
jsPsych.endCurrentTimeline()
}
}
}

var recall = {
type: 'digit-span-recall',
correct_order: function(){
return selection
},
trial_duration: recall_duration,
on_finish: function(){
acc = jsPsych.data.get().last(1).values()[0].accuracy;
if (acc==1){
if (highest_span_score < minSetSize){
highest_span_score = minSetSize
}
minSetSize+=1
nError = 0
} else if (nError < 1) { // checks for number of consecutive errors
nError += 1
} else {
if (consec_error_score < minSetSize){
consec_error_score = minSetSize
}
minSetSize = Math.max( 3, minSetSize-1)
}
if (minSetSize<=8){ // bottom code prevents immediate repition of digits
selection = jsPsych.randomization.sampleWithoutReplacement(possibleNumbers, minSetSize)
} else {
selection = jsPsych.randomization.sampleWithoutReplacement(possibleNumbers, 8)
var extra = minSetSize-8
var id = possibleNumbers.indexOf(selection[7])
possibleNumbers.splice(id, 1);
var extraNumbers= jsPsych.randomization.sampleWithoutReplacement(possibleNumbers, extra)
selection = selection.concat(extraNumbers)
possibleNumbers = ["1", "2", "3", "4", "5", "6", "7", "8", "9", "0"]
}
selection_id = -1
}
}

var feedback = {
type: 'html-keyboard-response',
stimulus: function(){
var text = ""
var accuracy = jsPsych.data.get().last(1).values()[0].accuracy
if (accuracy==1){
text += '

Correct

'
} else {
text += '

Incorrect

'
}
//text += '

New trial starting now.

'
return text
},
choices: jsPsych.NO_KEYS,
trial_duration: 1000
}

var conclusion = {
type: 'html-keyboard-response',
stimulus: function(){
return '

You have completed the task.
Thank you for your participation in this task.

Maximum number of digits recalled correctly was ' + highest_span_score +'.

Maximum number of digits reached before making two consecutive errors was ' +consec_error_score+'.

Please tell the Research Assistant you have completed the task.

'
},
choices: jsPsych.NO_KEYS
// trial_duration:1000
}

var test_stack = {
timeline: [test_stimuli],
repetitions: 17
}

var test_procedure = {
timeline: [test_stack, recall, feedback],
repetitions: nTrials
}

var demo_procedure = {
timeline: [test_stack, recall, feedback],
repetitions: 3
}

function saveData(filename, filedata){
$.ajax({
type:'post',
cache: false,
url: 'save_data.php', // this is the path to the above PHP script
data: {filename: filename, filedata: filedata}
});
};

var IDsub = Date.now()

var dataLog = {
type: 'html-keyboard-response',
stimulus: " ",
trial_duration: 100,
on_finish: function(data) {
var data = jsPsych.data.get().filter({trial_type: 'digit-span-recall'});
if (file_name == null){
file_name = "WM_digit_span_"+partN+"_"+IDsub.toString()+".csv"}
else{
file_name += ".csv"
}
if (local){
data.localSave('csv', file_name )
} else {
saveData(file_name, data.csv());
}
}
}

timeline = [p_details]
timeline.push({
type: 'fullscreen',
fullscreen_mode: true
});
timeline = timeline.concat([instructions, demo_procedure, instructions_node, test_procedure])
timeline.push({
type: 'fullscreen',
fullscreen_mode: false
});
timeline.push(dataLog)
timeline.push(conclusion)

jsPsych.init({
timeline: timeline,
on_finish: function() {
jsPsych.data.displayData(); // comment out if you do not want to display results at the end of task
}
});
</script>

[javidalpe]

Hi! Seems like your are trying to define your experiment using HTML instead of Javascript. The script you describe is HTML. The Javascript code is inside de <script> tags.

[ran-amram]

@javidalpe Such an incredible tool, huge respect! Definitely makes things a whole lot easier :)

There is one thing I'm having a bit of trouble with - and that's the CONDITION variable. How can I integrate it into my code?
I have two conditions in my experiment and I've set "Inter experiment conditions" to 2. I've defined an entire experimental timeline for each condition separately, so participants should be assigned to one of the conditions immediately upon entering the experimental environment.

I've tried several options incorporating the CONDITION variable. When previewing the experiment, each condition was consistently identified by setting a condition number only when using if statements:

However, in this case, the experiment itself (i.e., when entering through the provided url) always lands on the second condition. Could it be that the CONDITION variable is simply set at 2 due to the configuration of the "Inter experiment conditions"?

In any case, I've tried to go about it in different ways, and failed each time. I would love to know what it is I'm missing here and what is the proper way to use the CONDITION feature.

Many many thanks!

[amunn]

Hi @ran-amram are you sure your code doesn't have a line in which you actually set the CONDITION variable itself, perhaps added during testing? This might be overriding the setting that cognition.run assigns.

Unrelated, but do you really need two separate timelines for the different conditions? Usually you would use the CONDITION variable to set which set of simuli get presented, but the core timeline would be common.

[ran-amram]

@amunn Thank you for your reply!
Yep, definitely no such line. I did go there in some of my attempts, and found that it indeed did not work.

Good call about the separate timelines -- I'm in the midst of writing the code, and haven't decided yet how to go about it. This was just the way I defined it for the initial testing of this feature on Cognition.run.
In terms of writing the code itself, how would one use the CONDITION variable to do what you described? What separates the groups in my experiment are actually mostly the instructions rather than the stimuli, but I guess the same principle would apply?

[amunn]

@ran-amram

If it's just instructions that you need to have change, then I would just define two versions of your instructions HTML like this. This code works correctly for me on cognition.run

var timeline = [];
var v1_instructions_html = [ '<p> Version 1 instructions </p>' ];
var v2_instructions_html = ['<p> Version 2 instructions </p>' ];

var instructions = v1_instructions_html ;
  if (CONDITION == 2) {
     instructions = v2_instructions_html ;
     };
var instruction_screen = { 
      type: 'instructions',
      pages: instructions,
      show_clickable_nav: true,
      allow_backward: false
     };
timeline.push(instruction_screen);

jsPsych.init({
			timeline: timeline,
			});

[ran-amram]

@amunn
This worked perfectly!
Not sure I understand what I did wrong before, but now everything seems to be working :)

Huge thanks for your help!

[NoelijaW]

@javidalpe Hey! I have a question less focused on code, but rather on the specifics of cognition.run. At the moment we're running a Task where we shared the link directly on public platforms with potential participants. In that process many people just clicked the link and then closed the website, and a bot also opened it.
Because of that, we have around 39 finished trials and 160 unfinished ones in two days.

Is there a risk, if we continue that strategy and collect that many unfinished trials, that the site will crash? Is there a limit to the possible unfinished trials? Or should we duplicate our task and start not to share the link publicly?

I'd be grateful for any hints, and hope that the matter is comprehensible!

[kernel52]

@javidalpe, congratulations for the excellent work you did.
I find it very useful and I am interested in participating in the project (even for menial tasks as a wiki/how to writing, or maintaining some forum for related questions) if you ever want to open it to the community.

In the meantime, I would like to ask, if you could share a little bit more on the procedure behind the csv files, I'd like to know at what moment you do it (and how if possible) as I would be interested in doing some manipulations to the dataStructure (for a specific project) before saving the CSV.
Ideally I would like to be a able create two files per experiment (one with just personal info, and one with the results in a different format) but that might probably be quite complex as a task so I could pass if not possible.

thanks,
Nikolas

[Max-Lovell]

Hi, does anyone know of a way to host my study on cognition.run and then integrate this with a qualtrics survey?

[nrschoen]

Redirect links work great. You can implement a redirect from Qualtrics to cognition (also including URL parameters to keep track of subject ID’s). From cognition you can also redirect back to Qualtrics. Let me know if you can’t find a guide on Google. Cheers, Neil

…

On Wed, 28 Jul 2021 at 16:16, Max Lovell ***@***.***> wrote: Hi, does anyone know of a way to host my study on cognition.run and then integrate this with a qualtrics survey? — You are receiving this because you commented. Reply to this email directly, view it on GitHub <#754 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APRI3IQ3B2BMRX7DO4W72ZDT2AGMPANCNFSM4N3V7AUQ> .

[Max-Lovell]

Hi Neil - I'm looking into this approach so thanks for pointing it out - you or anyone stumbling across this comment might also be interested in this approach using an HTML iFrame to embed it straight into a question: https://community.qualtrics.com/XMcommunity/discussion/15331/redirects-to-external-sites-from-qualtrics-survey-and-back-again

[Max-Lovell]

I’m trying to embed my cognition.run experiment into a qualtrics survey with an HTML iFrame, as suggested here: https://community.qualtrics.com/XMcommunity/discussion/15331/redirects-to-external-sites-from-qualtrics-survey-and-back-again. This works using <iframe height=500 width=600 src=“https://iqpovrjfem.cognition.run”> to embed it in the page. However, when I try using the option to open it in a new tab with <a href=“https://iqpovrjfem.cognition.run” target="_blank"> Click Here for the Experiment</a> I get met with a page listing: Cannot GET /jfe/form/%E2%80%9Chttps:/iqpovrjfem.cognition.run%E2%80%9D. Does anyone know why this might be and how to fix it? Thanks!

[iholzleitner]

Loving Cognition.run and plan to use it for my student projects this year! Is there an easy way to batch delete stimuli?

[kernel52]

You can remove completely all the stimuli, but I wouldn't recommend it.

The only way I've found so far is to remove each one of them instead of saving.
see: #1972 (reply in thread)

[jodeleeuw]

Hi folks. I've added a cognition.run label to our discussions now. If you would like to post a question about cognition.run I recommend adding a new discussion topic instead of posting to this one, and we'll mark it with the appropriate label.