data-exercise-4.ipynb

{
 "cells": [
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "# Downloading the ABCD imaging data\n",
    "\n",
    "In the previous data exercises you have download and interacted with the [ABCD 3.0 release](https://nda.nih.gov/abcd/query/abcd-curated-annual-release-3.0.html). While there are many measures derived from the imaging data within the pre-packaged tabulated data, you may have noticed that the full set of MRI images are not included in this release.\n",
    "\n",
    "As stated on [NDA's website](https://nda.nih.gov/abcd/query/abcd-curated-annual-release-3.0.html): \n",
    "\n",
    "\"The raw MRI images and the minimally processed imaging files are over 100TB in size which may make data transfer difficult. \"\n",
    "\n",
    "The data are stored on [Amazon Simple Storage Service (s3)](https://docs.aws.amazon.com/AmazonS3/latest/userguide/Welcome.html) servers. \n",
    "\n",
    "There are multiple routes to obtaining the full imaging data, we'll focus on the following two:\n",
    "1. Using links from the [fmriresults01](https://nda.nih.gov/data_structure.html?short_name=fmriresults01) structure\n",
    "2. Using the [nda-abcd-s3-downloader](https://github.com/DCAN-Labs/nda-abcd-s3-downloader)\n",
    "\n",
    "Both routes involve creating a data package through the NDA, downloading a manifest file, parsing the manifest file, and finally downloading the data.\n",
    "\n",
    "For brevity, the exercises in this notebook will guide you through downloading the resting state and T1w data from 5 subjects using each of the above download methods. You will need active NDA credentials and an ABCD DUC to download the data.\n",
    "\n",
    "**A Note about GUIDs and BIDS**\n",
    "\n",
    "[From the NDA](https://nda.nih.gov/s/guid/nda-guid.html): \"The Global Unique Identifier (GUID) is a subject ID allowing researchers to share data specific to a study participant without exposing personally identifiable information (PII) and match participants across labs and research data repositories.\"\n",
    "\n",
    "The GUID's format is `NDAR_INVXXXXXXXX`, where `XXXXXXXX` is a random string of numbers and uppercase letters. The standard GUID format is *not* [BIDS compatible](https://bids-specification.readthedocs.io/en/stable/02-common-principles.html#file-name-structure). In BIDS, the underscore character is reserved to separate key:value entities (eg, `key1-value1_key2-value2`, `sub-01_task-rest`). For the BIDS imaging data on the NDA, the underscore in the GUID has been removed (ie, `NDARINVXXXXXXXX`), but be aware that you might need to do a string replace operation to remove the underscore from the GUIDs in the tabulated data to match the GUIDs in the BIDS imaging data."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "## Downloading the data using the fmriresults01 structure\n",
    "\n",
    "The general workflow on the NDA is to add data to your Filter Cart and then create a Data Package from the filter. Here we will create a Data Package from the *fmriresults01* data structure. See Getting Image Volumes [here](https://nda.nih.gov/abcd/query/abcd-release-faqs.html) for more info on the *fmriresults01* structure.\n",
    "\n",
    "**NOTE**: The `fmriresults01.txt` file is distributed in the ABCD 3.0 Release. So if you've already downnloaded that, then you could use that file. If so, you can skip to step 13.\n",
    "\n",
    "1. Let's begin at the [NDA's front page](https://nda.nih.gov/). Select **Get Data** > **Get Data**\n",
    "\n",
    "<img src=\"./screenshots/nda_frontpage.png\" width=\"900\" />\n",
    "\n",
    "***\n",
    "\n",
    "2. On the **NDA Query Tool**'s menu, select **Data Structures**. Then enter \"fmriresults01\" into the Text Search field and hit enter.\n",
    "\n",
    "<img src=\"./screenshots/nda_query.png\" width=\"900\" />\n",
    "\n",
    "***\n",
    "\n",
    "3. Click the **Processed MRI Data** link, which will open the structure. Then select **Add to Filter Cart** in the lower left corner.\n",
    "\n",
    "<img src=\"./screenshots/add_filter_cart.png\" width=\"150\" />\n",
    "\n",
    "Your Filter Cart will take a few minutes to update. Make yourself some tea. Once it is finished, you should see the following.\n",
    "\n",
    "<img src=\"./screenshots/filter_cart.png\" width=\"400\" />\n",
    "\n",
    "(Sample size may vary depending on when you are working through this exercise)\n",
    "\n",
    "***\n",
    "\n",
    "4. In the Filter Cart, select **Create Data Package/Add Data to Study**, which will take you to the Data Packaging Page.\n",
    "\n",
    "5. On the Data Packaging Page, select **Create Data Package**.\n",
    "\n",
    "<img src=\"./screenshots/create_data_package.png\" width=\"200\" />\n",
    "\n",
    "6. If you are not logged into the NDA, this will prompt you to log in with your credientials. After, you will see a menu to define your Data Package. Give it a short name and ensure that **Include Associated Data Files** is *unchecked*. Otherwise, the Data Package will contain the all images in *fmriresults*. It will be faster and more flexible to only download the pointers to the data and not the data istself. When you are finished entering this information, click **Create Data Package**.\n",
    "\n",
    "<img src=\"./screenshots/create_menu.png\" width=\"300\" />\n",
    "\n",
    "***\n",
    "\n",
    "7. You will see a window that confirms that your package was initiated. Click the link to navigate to your Dashboard.\n",
    "\n",
    "<img src=\"./screenshots/package_created.png\" width=\"350\" />\n",
    "\n",
    "***\n",
    "\n",
    "8. In the drop down menu on the Data Package Dashboard, select **My Data Packages**. You should see the Data Package you just created. It will take a few minutes to move from the \"Creating Package\" status to \"Ready to Download\". Maybe refill your tea. In the below image **ABCDndar** is the Data Package we just created. **ABCDdcan** will be created in the second section of this exercise.\n",
    "\n",
    "<img src=\"./screenshots/create_dash.png\" width=\"350\" />\n",
    "\n",
    "<img src=\"./screenshots/ready_dash.png\" width=\"350\" />\n",
    "\n",
    "***\n",
    "\n",
    "9. Once the Data Package is ready to download, we can use the [NDA tools](https://github.com/NDAR/nda-tools) to download it. The NDA tools are already installed and ready to use on the ABCD-ReproNim JupyterHub. The relevant command will be `downloadcmd`. Let's see what options `downloadcmd` has."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running NDATools Version 0.2.3\n",
      "Opening log: /home/jovyan/NDAValidationResults/debug_log_20210305T010219.txt\n",
      "usage: downloadcmd <S3_path_list>\n",
      "\n",
      "This application allows you to enter a list of aws S3 paths and will download\n",
      "the files to your drive in your home folder. Alternatively, you may enter a\n",
      "packageID, an NDA data structure file or a text file with s3 links, and the\n",
      "client will download all files from the S3 links listed. Please note, the\n",
      "maximum transfer limit of data is 5TB at one time.\n",
      "\n",
      "positional arguments:\n",
      "  <S3_path_list>        Will download all S3 files to your local drive\n",
      "\n",
      "optional arguments:\n",
      "  -h, --help            show this help message and exit\n",
      "  -dp, --package        Flags to download all S3 files in package.\n",
      "  -t, --txt             Flags that a text file has been entered from where to\n",
      "                        download S3 files.\n",
      "  -ds, --datastructure  Flags that a data structure text file has been entered\n",
      "                        from where to download S3 files.\n",
      "  -u <arg>, --username <arg>\n",
      "                        NDA username\n",
      "  -p <arg>, --password <arg>\n",
      "                        NDA password\n",
      "  -r <arg>, --resume <arg>\n",
      "                        Flags to restart a download process. If you already\n",
      "                        have some files downloaded, you must enter the\n",
      "                        directory where they are saved.\n",
      "  -d <arg>, --directory <arg>\n",
      "                        Enter an alternate full directory path where you would\n",
      "                        like your files to be saved.\n",
      "  -wt <arg>, --workerThreads <arg>\n",
      "                        Number of worker threads\n",
      "  -v, --verbose         Option to print out more detailed messages as the\n",
      "                        program runs.\n"
     ]
    }
   ],
   "source": [
    "! downloadcmd -h"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Recall that the `!` in the code cell of a Jupyter notebook means to execute that command using shell."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "10. Our first usage of `downloadcmd` will use the Package ID to download the associated package files. Let's put the ABCDndar package into it's own directory. If you have already set up your NDA credentials to download the ABCD 3.0 Release, then `downloadcmd` will use the already stored credentials.  "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mkdir: cannot create directory ‘/home/jovyan/ABCDndar3’: File exists\n",
      "Running NDATools Version 0.2.3\n",
      "Opening log: /home/jovyan/NDAValidationResults/debug_log_20210305T010225.txt\n"
     ]
    }
   ],
   "source": [
    "! mkdir /home/jovyan/ABCDndar3\n",
    "! downloadcmd 1186291 -dp -d /home/jovyan/ABCDndar3 # number is the data package ID from ndar"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "11. Once the download is complete, we can list the files. The relevant file is `fmriresults01.txt`, which contains information about each image in this structure."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "experiments\t   guid_pseudoguid.txt\tREADME.pdf\n",
      "fmriresults01.txt  package_info.txt\n"
     ]
    }
   ],
   "source": [
    "! ls /home/jovyan/ABCDndar3"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "12. `fmriresults01.txt` is a tab-separated table that contains information about corresponding image files. Let's read this table into python so that we can parse and choose only the image files we want."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "import pandas as pd\n",
    "fmri = pd.read_csv('/home/jovyan/ABCDndar3/fmriresults01.txt', sep='\\t', low_memory=False)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "*** \n",
    "\n",
    "13. Let's look at the structure and contents of the `fmriresults01.txt`. "
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>collection_id</th>\n",
       "      <th>fmriresults01_id</th>\n",
       "      <th>dataset_id</th>\n",
       "      <th>subjectkey</th>\n",
       "      <th>src_subject_id</th>\n",
       "      <th>origin_dataset_id</th>\n",
       "      <th>interview_date</th>\n",
       "      <th>interview_age</th>\n",
       "      <th>sex</th>\n",
       "      <th>experiment_id</th>\n",
       "      <th>...</th>\n",
       "      <th>qc_outcome</th>\n",
       "      <th>derived_files</th>\n",
       "      <th>scan_type</th>\n",
       "      <th>img03_id2</th>\n",
       "      <th>file_source2</th>\n",
       "      <th>session_det</th>\n",
       "      <th>image_history</th>\n",
       "      <th>manifest</th>\n",
       "      <th>image_description</th>\n",
       "      <th>collection_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>0</th>\n",
       "      <td>collection_id</td>\n",
       "      <td>fmriresults01_id</td>\n",
       "      <td>dataset_id</td>\n",
       "      <td>The NDAR Global Unique Identifier (GUID) for r...</td>\n",
       "      <td>Subject ID how it's defined in lab/project</td>\n",
       "      <td>Origin dataset Id</td>\n",
       "      <td>Date on which the interview/genetic test/sampl...</td>\n",
       "      <td>Age in months at the time of the interview/tes...</td>\n",
       "      <td>Sex of the subject</td>\n",
       "      <td>ID for the Experiment/settings/run</td>\n",
       "      <td>...</td>\n",
       "      <td>Provide information on the conclusion of the q...</td>\n",
       "      <td>An archive of the files produced by the pipeline</td>\n",
       "      <td>Type of Scan</td>\n",
       "      <td>Corresponds to row_id in image03 data structur...</td>\n",
       "      <td>File name/location, 2</td>\n",
       "      <td>session details</td>\n",
       "      <td>Image history,f.e. transformations steps and o...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Image description, i.e. DTI, fMRI, Fast SPGR, ...</td>\n",
       "      <td>collection_title</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2573</td>\n",
       "      <td>1140322</td>\n",
       "      <td>34648</td>\n",
       "      <td>NDAR_INVZVW3HKWN</td>\n",
       "      <td>NDAR_INVZVW3HKWN</td>\n",
       "      <td>17077</td>\n",
       "      <td>02/03/2017</td>\n",
       "      <td>131</td>\n",
       "      <td>M</td>\n",
       "      <td>650</td>\n",
       "      <td>...</td>\n",
       "      <td>pass</td>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVZV...</td>\n",
       "      <td>fMRI</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>ABCD-MPROC-SST</td>\n",
       "      <td>motion correction, B0 inhomogeneity correction...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Adolescent Brain Cognitive Development Study (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2573</td>\n",
       "      <td>1140323</td>\n",
       "      <td>34648</td>\n",
       "      <td>NDAR_INVXEYPED0K</td>\n",
       "      <td>NDAR_INVXEYPED0K</td>\n",
       "      <td>17049</td>\n",
       "      <td>11/29/2017</td>\n",
       "      <td>126</td>\n",
       "      <td>M</td>\n",
       "      <td>649</td>\n",
       "      <td>...</td>\n",
       "      <td>pass</td>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVXE...</td>\n",
       "      <td>fMRI</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>ABCD-MPROC-REST</td>\n",
       "      <td>motion correction, B0 inhomogeneity correction...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Adolescent Brain Cognitive Development Study (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2573</td>\n",
       "      <td>1140324</td>\n",
       "      <td>34648</td>\n",
       "      <td>NDAR_INVZ3638RYW</td>\n",
       "      <td>NDAR_INVZ3638RYW</td>\n",
       "      <td>17077</td>\n",
       "      <td>06/16/2017</td>\n",
       "      <td>125</td>\n",
       "      <td>M</td>\n",
       "      <td>649</td>\n",
       "      <td>...</td>\n",
       "      <td>pass</td>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVZ3...</td>\n",
       "      <td>fMRI</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>ABCD-MPROC-REST</td>\n",
       "      <td>motion correction, B0 inhomogeneity correction...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Adolescent Brain Cognitive Development Study (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2573</td>\n",
       "      <td>1140325</td>\n",
       "      <td>34648</td>\n",
       "      <td>NDAR_INVWVFZTJX6</td>\n",
       "      <td>NDAR_INVWVFZTJX6</td>\n",
       "      <td>24209</td>\n",
       "      <td>05/21/2019</td>\n",
       "      <td>147</td>\n",
       "      <td>M</td>\n",
       "      <td>651</td>\n",
       "      <td>...</td>\n",
       "      <td>pass</td>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVWV...</td>\n",
       "      <td>fMRI</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>ABCD-MPROC-NBACK</td>\n",
       "      <td>motion correction, B0 inhomogeneity correction...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Adolescent Brain Cognitive Development Study (...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "   collection_id  fmriresults01_id  dataset_id  \\\n",
       "0  collection_id  fmriresults01_id  dataset_id   \n",
       "1           2573           1140322       34648   \n",
       "2           2573           1140323       34648   \n",
       "3           2573           1140324       34648   \n",
       "4           2573           1140325       34648   \n",
       "\n",
       "                                          subjectkey  \\\n",
       "0  The NDAR Global Unique Identifier (GUID) for r...   \n",
       "1                                   NDAR_INVZVW3HKWN   \n",
       "2                                   NDAR_INVXEYPED0K   \n",
       "3                                   NDAR_INVZ3638RYW   \n",
       "4                                   NDAR_INVWVFZTJX6   \n",
       "\n",
       "                               src_subject_id  origin_dataset_id  \\\n",
       "0  Subject ID how it's defined in lab/project  Origin dataset Id   \n",
       "1                            NDAR_INVZVW3HKWN              17077   \n",
       "2                            NDAR_INVXEYPED0K              17049   \n",
       "3                            NDAR_INVZ3638RYW              17077   \n",
       "4                            NDAR_INVWVFZTJX6              24209   \n",
       "\n",
       "                                      interview_date  \\\n",
       "0  Date on which the interview/genetic test/sampl...   \n",
       "1                                         02/03/2017   \n",
       "2                                         11/29/2017   \n",
       "3                                         06/16/2017   \n",
       "4                                         05/21/2019   \n",
       "\n",
       "                                       interview_age                 sex  \\\n",
       "0  Age in months at the time of the interview/tes...  Sex of the subject   \n",
       "1                                                131                   M   \n",
       "2                                                126                   M   \n",
       "3                                                125                   M   \n",
       "4                                                147                   M   \n",
       "\n",
       "                        experiment_id  ...  \\\n",
       "0  ID for the Experiment/settings/run  ...   \n",
       "1                                 650  ...   \n",
       "2                                 649  ...   \n",
       "3                                 649  ...   \n",
       "4                                 651  ...   \n",
       "\n",
       "                                          qc_outcome  \\\n",
       "0  Provide information on the conclusion of the q...   \n",
       "1                                               pass   \n",
       "2                                               pass   \n",
       "3                                               pass   \n",
       "4                                               pass   \n",
       "\n",
       "                                       derived_files     scan_type  \\\n",
       "0   An archive of the files produced by the pipeline  Type of Scan   \n",
       "1  s3://NDAR_Central_4/submission_32739/NDARINVZV...          fMRI   \n",
       "2  s3://NDAR_Central_4/submission_32739/NDARINVXE...          fMRI   \n",
       "3  s3://NDAR_Central_4/submission_32739/NDARINVZ3...          fMRI   \n",
       "4  s3://NDAR_Central_4/submission_32739/NDARINVWV...          fMRI   \n",
       "\n",
       "                                           img03_id2           file_source2  \\\n",
       "0  Corresponds to row_id in image03 data structur...  File name/location, 2   \n",
       "1                                                NaN                    NaN   \n",
       "2                                                NaN                    NaN   \n",
       "3                                                NaN                    NaN   \n",
       "4                                                NaN                    NaN   \n",
       "\n",
       "        session_det                                      image_history  \\\n",
       "0   session details  Image history,f.e. transformations steps and o...   \n",
       "1    ABCD-MPROC-SST  motion correction, B0 inhomogeneity correction...   \n",
       "2   ABCD-MPROC-REST  motion correction, B0 inhomogeneity correction...   \n",
       "3   ABCD-MPROC-REST  motion correction, B0 inhomogeneity correction...   \n",
       "4  ABCD-MPROC-NBACK  motion correction, B0 inhomogeneity correction...   \n",
       "\n",
       "  manifest                                  image_description  \\\n",
       "0      NaN  Image description, i.e. DTI, fMRI, Fast SPGR, ...   \n",
       "1      NaN                                                NaN   \n",
       "2      NaN                                                NaN   \n",
       "3      NaN                                                NaN   \n",
       "4      NaN                                                NaN   \n",
       "\n",
       "                                    collection_title  \n",
       "0                                   collection_title  \n",
       "1  Adolescent Brain Cognitive Development Study (...  \n",
       "2  Adolescent Brain Cognitive Development Study (...  \n",
       "3  Adolescent Brain Cognitive Development Study (...  \n",
       "4  Adolescent Brain Cognitive Development Study (...  \n",
       "\n",
       "[5 rows x 32 columns]"
      ]
     },
     "execution_count": 6,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fmri.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "We can see that the first row contains a detailed description of the column. We won't need to include this in our parsing, so we can drop it."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 7,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>collection_id</th>\n",
       "      <th>fmriresults01_id</th>\n",
       "      <th>dataset_id</th>\n",
       "      <th>subjectkey</th>\n",
       "      <th>src_subject_id</th>\n",
       "      <th>origin_dataset_id</th>\n",
       "      <th>interview_date</th>\n",
       "      <th>interview_age</th>\n",
       "      <th>sex</th>\n",
       "      <th>experiment_id</th>\n",
       "      <th>...</th>\n",
       "      <th>qc_outcome</th>\n",
       "      <th>derived_files</th>\n",
       "      <th>scan_type</th>\n",
       "      <th>img03_id2</th>\n",
       "      <th>file_source2</th>\n",
       "      <th>session_det</th>\n",
       "      <th>image_history</th>\n",
       "      <th>manifest</th>\n",
       "      <th>image_description</th>\n",
       "      <th>collection_title</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>2573</td>\n",
       "      <td>1140322</td>\n",
       "      <td>34648</td>\n",
       "      <td>NDAR_INVZVW3HKWN</td>\n",
       "      <td>NDAR_INVZVW3HKWN</td>\n",
       "      <td>17077</td>\n",
       "      <td>02/03/2017</td>\n",
       "      <td>131</td>\n",
       "      <td>M</td>\n",
       "      <td>650</td>\n",
       "      <td>...</td>\n",
       "      <td>pass</td>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVZV...</td>\n",
       "      <td>fMRI</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>ABCD-MPROC-SST</td>\n",
       "      <td>motion correction, B0 inhomogeneity correction...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Adolescent Brain Cognitive Development Study (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>2573</td>\n",
       "      <td>1140323</td>\n",
       "      <td>34648</td>\n",
       "      <td>NDAR_INVXEYPED0K</td>\n",
       "      <td>NDAR_INVXEYPED0K</td>\n",
       "      <td>17049</td>\n",
       "      <td>11/29/2017</td>\n",
       "      <td>126</td>\n",
       "      <td>M</td>\n",
       "      <td>649</td>\n",
       "      <td>...</td>\n",
       "      <td>pass</td>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVXE...</td>\n",
       "      <td>fMRI</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>ABCD-MPROC-REST</td>\n",
       "      <td>motion correction, B0 inhomogeneity correction...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Adolescent Brain Cognitive Development Study (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>2573</td>\n",
       "      <td>1140324</td>\n",
       "      <td>34648</td>\n",
       "      <td>NDAR_INVZ3638RYW</td>\n",
       "      <td>NDAR_INVZ3638RYW</td>\n",
       "      <td>17077</td>\n",
       "      <td>06/16/2017</td>\n",
       "      <td>125</td>\n",
       "      <td>M</td>\n",
       "      <td>649</td>\n",
       "      <td>...</td>\n",
       "      <td>pass</td>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVZ3...</td>\n",
       "      <td>fMRI</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>ABCD-MPROC-REST</td>\n",
       "      <td>motion correction, B0 inhomogeneity correction...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Adolescent Brain Cognitive Development Study (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>2573</td>\n",
       "      <td>1140325</td>\n",
       "      <td>34648</td>\n",
       "      <td>NDAR_INVWVFZTJX6</td>\n",
       "      <td>NDAR_INVWVFZTJX6</td>\n",
       "      <td>24209</td>\n",
       "      <td>05/21/2019</td>\n",
       "      <td>147</td>\n",
       "      <td>M</td>\n",
       "      <td>651</td>\n",
       "      <td>...</td>\n",
       "      <td>pass</td>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVWV...</td>\n",
       "      <td>fMRI</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>ABCD-MPROC-NBACK</td>\n",
       "      <td>motion correction, B0 inhomogeneity correction...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Adolescent Brain Cognitive Development Study (...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>2573</td>\n",
       "      <td>1140326</td>\n",
       "      <td>34648</td>\n",
       "      <td>NDAR_INVXFURZ24F</td>\n",
       "      <td>NDAR_INVXFURZ24F</td>\n",
       "      <td>17050</td>\n",
       "      <td>01/08/2018</td>\n",
       "      <td>109</td>\n",
       "      <td>M</td>\n",
       "      <td>649</td>\n",
       "      <td>...</td>\n",
       "      <td>pass</td>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVXF...</td>\n",
       "      <td>fMRI</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>ABCD-MPROC-REST</td>\n",
       "      <td>motion correction, B0 inhomogeneity correction...</td>\n",
       "      <td>NaN</td>\n",
       "      <td>NaN</td>\n",
       "      <td>Adolescent Brain Cognitive Development Study (...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>5 rows × 32 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "  collection_id fmriresults01_id dataset_id        subjectkey  \\\n",
       "1          2573          1140322      34648  NDAR_INVZVW3HKWN   \n",
       "2          2573          1140323      34648  NDAR_INVXEYPED0K   \n",
       "3          2573          1140324      34648  NDAR_INVZ3638RYW   \n",
       "4          2573          1140325      34648  NDAR_INVWVFZTJX6   \n",
       "5          2573          1140326      34648  NDAR_INVXFURZ24F   \n",
       "\n",
       "     src_subject_id origin_dataset_id interview_date interview_age sex  \\\n",
       "1  NDAR_INVZVW3HKWN             17077     02/03/2017           131   M   \n",
       "2  NDAR_INVXEYPED0K             17049     11/29/2017           126   M   \n",
       "3  NDAR_INVZ3638RYW             17077     06/16/2017           125   M   \n",
       "4  NDAR_INVWVFZTJX6             24209     05/21/2019           147   M   \n",
       "5  NDAR_INVXFURZ24F             17050     01/08/2018           109   M   \n",
       "\n",
       "  experiment_id  ... qc_outcome  \\\n",
       "1           650  ...       pass   \n",
       "2           649  ...       pass   \n",
       "3           649  ...       pass   \n",
       "4           651  ...       pass   \n",
       "5           649  ...       pass   \n",
       "\n",
       "                                       derived_files scan_type img03_id2  \\\n",
       "1  s3://NDAR_Central_4/submission_32739/NDARINVZV...      fMRI       NaN   \n",
       "2  s3://NDAR_Central_4/submission_32739/NDARINVXE...      fMRI       NaN   \n",
       "3  s3://NDAR_Central_4/submission_32739/NDARINVZ3...      fMRI       NaN   \n",
       "4  s3://NDAR_Central_4/submission_32739/NDARINVWV...      fMRI       NaN   \n",
       "5  s3://NDAR_Central_4/submission_32739/NDARINVXF...      fMRI       NaN   \n",
       "\n",
       "  file_source2       session_det  \\\n",
       "1          NaN    ABCD-MPROC-SST   \n",
       "2          NaN   ABCD-MPROC-REST   \n",
       "3          NaN   ABCD-MPROC-REST   \n",
       "4          NaN  ABCD-MPROC-NBACK   \n",
       "5          NaN   ABCD-MPROC-REST   \n",
       "\n",
       "                                       image_history manifest  \\\n",
       "1  motion correction, B0 inhomogeneity correction...      NaN   \n",
       "2  motion correction, B0 inhomogeneity correction...      NaN   \n",
       "3  motion correction, B0 inhomogeneity correction...      NaN   \n",
       "4  motion correction, B0 inhomogeneity correction...      NaN   \n",
       "5  motion correction, B0 inhomogeneity correction...      NaN   \n",
       "\n",
       "  image_description                                   collection_title  \n",
       "1               NaN  Adolescent Brain Cognitive Development Study (...  \n",
       "2               NaN  Adolescent Brain Cognitive Development Study (...  \n",
       "3               NaN  Adolescent Brain Cognitive Development Study (...  \n",
       "4               NaN  Adolescent Brain Cognitive Development Study (...  \n",
       "5               NaN  Adolescent Brain Cognitive Development Study (...  \n",
       "\n",
       "[5 rows x 32 columns]"
      ]
     },
     "execution_count": 7,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "fmri = fmri.drop([0])\n",
    "fmri.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "14. We do not need most of the information in this table. The relevant columns are `file_source`, which contains the s3 links to the raw DICOM images and `derived_files` which contains the s3 links to the minimally preprocessed images. Here we will focus on downloading the minimally processed files in `derived_files`. We will create a dataframe that contains the s3 links and other relevant fields so that we can filter the s3 links. Explore other columns of this table to see the processing steps that has been applied to the `derived_files`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>derived_files</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVZV...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVXE...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVZ3...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVWV...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVXF...</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                       derived_files\n",
       "1  s3://NDAR_Central_4/submission_32739/NDARINVZV...\n",
       "2  s3://NDAR_Central_4/submission_32739/NDARINVXE...\n",
       "3  s3://NDAR_Central_4/submission_32739/NDARINVZ3...\n",
       "4  s3://NDAR_Central_4/submission_32739/NDARINVWV...\n",
       "5  s3://NDAR_Central_4/submission_32739/NDARINVXF..."
      ]
     },
     "execution_count": 8,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "# create a new data frame from the derived_files column\n",
    "s3_derv = fmri.loc[:,['derived_files']] \n",
    "# view dataframe\n",
    "s3_derv.head()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the format of the s3 links to see how we could parse this for filtering:\n",
    "\n",
    "*s3://NDAR_Central_4/submission_32739/NDARINVXXXXXXXX_baselineYear1Arm1_ABCD-MPROC-SST-fMRI_XXXXXXXXXXXXXX.tgz*\n",
    "\n",
    "- *s3://NDAR_Central_4/submission_32739* is the location of the data on the s3 server\n",
    "- *NDARINVXXXXXXXX* is the GUID\n",
    "- *baselineYear1Arm1* is the session\n",
    "- *ABCD-MPROC-SST-fMRI* is the scan type information\n",
    "- The number at the end of the file is the acqusition date/time\n",
    "- *.tgz* is the TAR archive file extension\n",
    "\n",
    "We can use python's ability to split strings to parse these strings so that we can filter by GUID, session, and scan type. Let's see an example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "['s3:',\n",
       " '',\n",
       " 'NDAR_Central_4',\n",
       " 'submission_32739',\n",
       " 'NDARINVXXXXXXXX_baselineYear1Arm1_ABCD-MPROC-SST-fMRI_XXXXXXXXXXXXXX.tgz']"
      ]
     },
     "execution_count": 9,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "example = 's3://NDAR_Central_4/submission_32739/NDARINVXXXXXXXX_baselineYear1Arm1_ABCD-MPROC-SST-fMRI_XXXXXXXXXXXXXX.tgz'\n",
    "example.split('/')"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above code splits the `example` string into a list of strings at every occurence of `/`.\n",
    "\n",
    "`.split` only operates on strings, but we have an entire column of strings we want to split. Here we can leverage python's list comprehension to iterate through each string.\n",
    "\n",
    "For example:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 10,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "[['s3:',\n",
       "  '',\n",
       "  'NDAR_Central_4',\n",
       "  'submission_32739',\n",
       "  'NDARINVZVW3HKWN_baselineYear1Arm1_ABCD-MPROC-SST-fMRI_20170203160134.tgz'],\n",
       " ['s3:',\n",
       "  '',\n",
       "  'NDAR_Central_4',\n",
       "  'submission_32739',\n",
       "  'NDARINVXEYPED0K_baselineYear1Arm1_ABCD-MPROC-rsfMRI_20171129140732.tgz'],\n",
       " ['s3:',\n",
       "  '',\n",
       "  'NDAR_Central_4',\n",
       "  'submission_32739',\n",
       "  'NDARINVZ3638RYW_baselineYear1Arm1_ABCD-MPROC-rsfMRI_20170709114733.tgz']]"
      ]
     },
     "execution_count": 10,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "test_split = [i.split('/') for i in s3_derv['derived_files']]\n",
    "test_split[0:3]"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above code submits the same split operation to every item in the `s3_derv['derived_files']` data frame we created above. You could also complete this with a regular `for` loop, but list comprehension is cleaner and more efficient.\n",
    "\n",
    "We can leverage string splitting and list comprehension to parse each s3 link into a corresponding GUID, session, and scan type."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 13,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>derived_files</th>\n",
       "      <th>guid</th>\n",
       "      <th>session</th>\n",
       "      <th>scan</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>1</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVZV...</td>\n",
       "      <td>NDARINVZVW3HKWN</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-SST-fMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVXE...</td>\n",
       "      <td>NDARINVXEYPED0K</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVZ3...</td>\n",
       "      <td>NDARINVZ3638RYW</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>4</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVWV...</td>\n",
       "      <td>NDARINVWVFZTJX6</td>\n",
       "      <td>2YearFollowUpYArm1</td>\n",
       "      <td>MPROC-nBack-fMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVXF...</td>\n",
       "      <td>NDARINVXFURZ24F</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>...</th>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "      <td>...</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200204</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINV18...</td>\n",
       "      <td>NDARINV18KYVZ3C</td>\n",
       "      <td>2YearFollowUpYArm1</td>\n",
       "      <td>MPROC-nBack-fMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200205</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINV1K...</td>\n",
       "      <td>NDARINV1KZTEZF5</td>\n",
       "      <td>2YearFollowUpYArm1</td>\n",
       "      <td>MPROC-nBack-fMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200206</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINV0P...</td>\n",
       "      <td>NDARINV0PKCP1P3</td>\n",
       "      <td>2YearFollowUpYArm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200207</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINV27...</td>\n",
       "      <td>NDARINV279N3WC9</td>\n",
       "      <td>2YearFollowUpYArm1</td>\n",
       "      <td>MPROC-SST-fMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>200208</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINV1N...</td>\n",
       "      <td>NDARINV1NJK6BVY</td>\n",
       "      <td>2YearFollowUpYArm1</td>\n",
       "      <td>MPROC-MID-fMRI</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "<p>200208 rows × 4 columns</p>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            derived_files             guid  \\\n",
       "1       s3://NDAR_Central_4/submission_32739/NDARINVZV...  NDARINVZVW3HKWN   \n",
       "2       s3://NDAR_Central_4/submission_32739/NDARINVXE...  NDARINVXEYPED0K   \n",
       "3       s3://NDAR_Central_4/submission_32739/NDARINVZ3...  NDARINVZ3638RYW   \n",
       "4       s3://NDAR_Central_4/submission_32739/NDARINVWV...  NDARINVWVFZTJX6   \n",
       "5       s3://NDAR_Central_4/submission_32739/NDARINVXF...  NDARINVXFURZ24F   \n",
       "...                                                   ...              ...   \n",
       "200204  s3://NDAR_Central_4/submission_32739/NDARINV18...  NDARINV18KYVZ3C   \n",
       "200205  s3://NDAR_Central_4/submission_32739/NDARINV1K...  NDARINV1KZTEZF5   \n",
       "200206  s3://NDAR_Central_4/submission_32739/NDARINV0P...  NDARINV0PKCP1P3   \n",
       "200207  s3://NDAR_Central_4/submission_32739/NDARINV27...  NDARINV279N3WC9   \n",
       "200208  s3://NDAR_Central_4/submission_32739/NDARINV1N...  NDARINV1NJK6BVY   \n",
       "\n",
       "                   session              scan  \n",
       "1        baselineYear1Arm1    MPROC-SST-fMRI  \n",
       "2        baselineYear1Arm1      MPROC-rsfMRI  \n",
       "3        baselineYear1Arm1      MPROC-rsfMRI  \n",
       "4       2YearFollowUpYArm1  MPROC-nBack-fMRI  \n",
       "5        baselineYear1Arm1      MPROC-rsfMRI  \n",
       "...                    ...               ...  \n",
       "200204  2YearFollowUpYArm1  MPROC-nBack-fMRI  \n",
       "200205  2YearFollowUpYArm1  MPROC-nBack-fMRI  \n",
       "200206  2YearFollowUpYArm1      MPROC-rsfMRI  \n",
       "200207  2YearFollowUpYArm1    MPROC-SST-fMRI  \n",
       "200208  2YearFollowUpYArm1    MPROC-MID-fMRI  \n",
       "\n",
       "[200208 rows x 4 columns]"
      ]
     },
     "execution_count": 13,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s3_derv['guid'] = [i.split('/')[-1].split('_')[0] for i in s3_derv['derived_files']] # get the GUID\n",
    "s3_derv['session'] = [i.split('/')[-1].split('_')[1] for i in s3_derv['derived_files']] # get the session\n",
    "s3_derv['scan'] = [i.split('/')[-1].split('_')[2].split('-',1)[-1] for i in s3_derv['derived_files']] # get the scan type\n",
    "\n",
    "s3_derv"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "The above list comprehension and string splitting code looks complicated, let's break down the code for parsing the scan type:\n",
    "\n",
    "- `[i for i in s3_derv['derived_files']` is looping through every string in `s3_derv['derived_files']`. `i` will be the string in the current iteration.\n",
    "- `i.split('/')[-1]` gives us the last (`[-1]`) item in the list (the filename) once you split the full s3 link by the `/` character.\n",
    "- The second `.split('_')[2]` splits the filename by `_`. `[2]` is choosing the third item in that list (because of 0 indexing). This is `ABCD-MPROC-SST-fMRI` in the above example.\n",
    "- The third `.split('-',1)[-1]` is splitting `ABCD-MPROC-SST-fMRI` by `-`, only by the first occurence of `-`. `[-1]` means that we are grabbing the last in that two item list (`MPROC-SST-fMRI`)."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now we have a dataframe where we can filter s3 links by GUID, session, and scan type! Let's see what scan types we have:"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 12,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MPROC-rsfMRI        62398\n",
       "MPROC-MID-fMRI      28864\n",
       "MPROC-nBack-fMRI    28331\n",
       "MPROC-SST-fMRI      28231\n",
       "MPROC-DTI           18589\n",
       "MPROC-T1            17349\n",
       "MPROC-T2            16446\n",
       "Name: scan, dtype: int64"
      ]
     },
     "execution_count": 12,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "s3_derv['scan'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "15. Let's specify our filtering critera. Choose 5 subject GUIDs (you can choose 5 random GUIDs from your work on the ABCD 3.0 Release), only the `baselineYear1Arm1` session, and scan types of `MPROC-T1` and `MPROC-rsfMRI`."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 14,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/html": [
       "<div>\n",
       "<style scoped>\n",
       "    .dataframe tbody tr th:only-of-type {\n",
       "        vertical-align: middle;\n",
       "    }\n",
       "\n",
       "    .dataframe tbody tr th {\n",
       "        vertical-align: top;\n",
       "    }\n",
       "\n",
       "    .dataframe thead th {\n",
       "        text-align: right;\n",
       "    }\n",
       "</style>\n",
       "<table border=\"1\" class=\"dataframe\">\n",
       "  <thead>\n",
       "    <tr style=\"text-align: right;\">\n",
       "      <th></th>\n",
       "      <th>derived_files</th>\n",
       "      <th>guid</th>\n",
       "      <th>session</th>\n",
       "      <th>scan</th>\n",
       "    </tr>\n",
       "  </thead>\n",
       "  <tbody>\n",
       "    <tr>\n",
       "      <th>183141</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINV0P...</td>\n",
       "      <td>NDARINV0PKCP1P3</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-T1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>193027</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINV0P...</td>\n",
       "      <td>NDARINV0PKCP1P3</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>193029</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINV0P...</td>\n",
       "      <td>NDARINV0PKCP1P3</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>195322</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINV0P...</td>\n",
       "      <td>NDARINV0PKCP1P3</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>197927</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINV0P...</td>\n",
       "      <td>NDARINV0PKCP1P3</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>164645</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVXE...</td>\n",
       "      <td>NDARINVXEYPED0K</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-T1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>2</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVXE...</td>\n",
       "      <td>NDARINVXEYPED0K</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3238</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVXE...</td>\n",
       "      <td>NDARINVXEYPED0K</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3878</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVXE...</td>\n",
       "      <td>NDARINVXEYPED0K</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3880</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVXE...</td>\n",
       "      <td>NDARINVXEYPED0K</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>164647</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVXF...</td>\n",
       "      <td>NDARINVXFURZ24F</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-T1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>5</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVXF...</td>\n",
       "      <td>NDARINVXFURZ24F</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>8</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVXF...</td>\n",
       "      <td>NDARINVXFURZ24F</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>11</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVXF...</td>\n",
       "      <td>NDARINVXFURZ24F</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6525</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVXF...</td>\n",
       "      <td>NDARINVXFURZ24F</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>179640</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVZ3...</td>\n",
       "      <td>NDARINVZ3638RYW</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-T1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVZ3...</td>\n",
       "      <td>NDARINVZ3638RYW</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>6</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVZ3...</td>\n",
       "      <td>NDARINVZ3638RYW</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>9</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVZ3...</td>\n",
       "      <td>NDARINVZ3638RYW</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>3882</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVZ3...</td>\n",
       "      <td>NDARINVZ3638RYW</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178927</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVZV...</td>\n",
       "      <td>NDARINVZVW3HKWN</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-T1</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>176539</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVZV...</td>\n",
       "      <td>NDARINVZVW3HKWN</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>176540</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVZV...</td>\n",
       "      <td>NDARINVZVW3HKWN</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178393</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVZV...</td>\n",
       "      <td>NDARINVZVW3HKWN</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "    <tr>\n",
       "      <th>178394</th>\n",
       "      <td>s3://NDAR_Central_4/submission_32739/NDARINVZV...</td>\n",
       "      <td>NDARINVZVW3HKWN</td>\n",
       "      <td>baselineYear1Arm1</td>\n",
       "      <td>MPROC-rsfMRI</td>\n",
       "    </tr>\n",
       "  </tbody>\n",
       "</table>\n",
       "</div>"
      ],
      "text/plain": [
       "                                            derived_files             guid  \\\n",
       "183141  s3://NDAR_Central_4/submission_32739/NDARINV0P...  NDARINV0PKCP1P3   \n",
       "193027  s3://NDAR_Central_4/submission_32739/NDARINV0P...  NDARINV0PKCP1P3   \n",
       "193029  s3://NDAR_Central_4/submission_32739/NDARINV0P...  NDARINV0PKCP1P3   \n",
       "195322  s3://NDAR_Central_4/submission_32739/NDARINV0P...  NDARINV0PKCP1P3   \n",
       "197927  s3://NDAR_Central_4/submission_32739/NDARINV0P...  NDARINV0PKCP1P3   \n",
       "164645  s3://NDAR_Central_4/submission_32739/NDARINVXE...  NDARINVXEYPED0K   \n",
       "2       s3://NDAR_Central_4/submission_32739/NDARINVXE...  NDARINVXEYPED0K   \n",
       "3238    s3://NDAR_Central_4/submission_32739/NDARINVXE...  NDARINVXEYPED0K   \n",
       "3878    s3://NDAR_Central_4/submission_32739/NDARINVXE...  NDARINVXEYPED0K   \n",
       "3880    s3://NDAR_Central_4/submission_32739/NDARINVXE...  NDARINVXEYPED0K   \n",
       "164647  s3://NDAR_Central_4/submission_32739/NDARINVXF...  NDARINVXFURZ24F   \n",
       "5       s3://NDAR_Central_4/submission_32739/NDARINVXF...  NDARINVXFURZ24F   \n",
       "8       s3://NDAR_Central_4/submission_32739/NDARINVXF...  NDARINVXFURZ24F   \n",
       "11      s3://NDAR_Central_4/submission_32739/NDARINVXF...  NDARINVXFURZ24F   \n",
       "6525    s3://NDAR_Central_4/submission_32739/NDARINVXF...  NDARINVXFURZ24F   \n",
       "179640  s3://NDAR_Central_4/submission_32739/NDARINVZ3...  NDARINVZ3638RYW   \n",
       "3       s3://NDAR_Central_4/submission_32739/NDARINVZ3...  NDARINVZ3638RYW   \n",
       "6       s3://NDAR_Central_4/submission_32739/NDARINVZ3...  NDARINVZ3638RYW   \n",
       "9       s3://NDAR_Central_4/submission_32739/NDARINVZ3...  NDARINVZ3638RYW   \n",
       "3882    s3://NDAR_Central_4/submission_32739/NDARINVZ3...  NDARINVZ3638RYW   \n",
       "178927  s3://NDAR_Central_4/submission_32739/NDARINVZV...  NDARINVZVW3HKWN   \n",
       "176539  s3://NDAR_Central_4/submission_32739/NDARINVZV...  NDARINVZVW3HKWN   \n",
       "176540  s3://NDAR_Central_4/submission_32739/NDARINVZV...  NDARINVZVW3HKWN   \n",
       "178393  s3://NDAR_Central_4/submission_32739/NDARINVZV...  NDARINVZVW3HKWN   \n",
       "178394  s3://NDAR_Central_4/submission_32739/NDARINVZV...  NDARINVZVW3HKWN   \n",
       "\n",
       "                  session          scan  \n",
       "183141  baselineYear1Arm1      MPROC-T1  \n",
       "193027  baselineYear1Arm1  MPROC-rsfMRI  \n",
       "193029  baselineYear1Arm1  MPROC-rsfMRI  \n",
       "195322  baselineYear1Arm1  MPROC-rsfMRI  \n",
       "197927  baselineYear1Arm1  MPROC-rsfMRI  \n",
       "164645  baselineYear1Arm1      MPROC-T1  \n",
       "2       baselineYear1Arm1  MPROC-rsfMRI  \n",
       "3238    baselineYear1Arm1  MPROC-rsfMRI  \n",
       "3878    baselineYear1Arm1  MPROC-rsfMRI  \n",
       "3880    baselineYear1Arm1  MPROC-rsfMRI  \n",
       "164647  baselineYear1Arm1      MPROC-T1  \n",
       "5       baselineYear1Arm1  MPROC-rsfMRI  \n",
       "8       baselineYear1Arm1  MPROC-rsfMRI  \n",
       "11      baselineYear1Arm1  MPROC-rsfMRI  \n",
       "6525    baselineYear1Arm1  MPROC-rsfMRI  \n",
       "179640  baselineYear1Arm1      MPROC-T1  \n",
       "3       baselineYear1Arm1  MPROC-rsfMRI  \n",
       "6       baselineYear1Arm1  MPROC-rsfMRI  \n",
       "9       baselineYear1Arm1  MPROC-rsfMRI  \n",
       "3882    baselineYear1Arm1  MPROC-rsfMRI  \n",
       "178927  baselineYear1Arm1      MPROC-T1  \n",
       "176539  baselineYear1Arm1  MPROC-rsfMRI  \n",
       "176540  baselineYear1Arm1  MPROC-rsfMRI  \n",
       "178393  baselineYear1Arm1  MPROC-rsfMRI  \n",
       "178394  baselineYear1Arm1  MPROC-rsfMRI  "
      ]
     },
     "execution_count": 14,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "subjs = ['NDARINVZVW3HKWN', 'NDARINVXEYPED0K', 'NDARINVZ3638RYW', \n",
    "        'NDARINVXFURZ24F', 'NDARINV0PKCP1P3'] # enter 5 GUIDs.\n",
    "runs = ['MPROC-T1', 'MPROC-rsfMRI'] # need to match the scan types in s3_derv\n",
    "ses = ['baselineYear1Arm1'] # session\n",
    "\n",
    "# filter the s3_derv data frame using the above filters\n",
    "sub_s3derv = s3_derv[s3_derv['guid'].isin(subjs) & s3_derv['scan'].isin(runs) & s3_derv['session'].isin(ses)]\n",
    "sub_s3derv.sort_values(['guid', 'scan']) # sort to make it pretty"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's see a count of how many s3 links met the filter criteria."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 25,
   "metadata": {},
   "outputs": [
    {
     "data": {
      "text/plain": [
       "MPROC-rsfMRI    20\n",
       "MPROC-T1         5\n",
       "Name: scan, dtype: int64"
      ]
     },
     "execution_count": 25,
     "metadata": {},
     "output_type": "execute_result"
    }
   ],
   "source": [
    "sub_s3derv['scan'].value_counts()"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "16. Great! Now we can write the filtered s3 links to a text file. `s3_derv_links_5subj.txt` will be a simple text file that only contains the relevant s3 links."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 26,
   "metadata": {},
   "outputs": [],
   "source": [
    "with open('/home/jovyan/ABCDndar3/s3_derv_links_5subj.txt', 'w') as f:\n",
    "    f.write('\\n'.join(sub_s3derv['derived_files']))"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "17. Now we can use `downloadcmd` to download the actual data! Let's also make a directory to store the downloaded files. The download will take a few minutes. Brew some more tea."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 27,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "mkdir: cannot create directory ‘/home/jovyan/ABCDndar3/tar_files’: File exists\n",
      "Running NDATools Version 0.2.3\n",
      "Opening log: /home/jovyan/NDAValidationResults/debug_log_20210305T011756.txt\n"
     ]
    }
   ],
   "source": [
    "! mkdir /home/jovyan/ABCDndar3/tar_files\n",
    "! downloadcmd -d /home/jovyan/ABCDndar3/tar_files -t /home/jovyan/ABCDndar3/s3_derv_links_5subj.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "18. Let's list out the files we've downloaded. You'll notice that the data was downloaded into a `submission_XXXXX` directory. You will also notice that the files are in `.tgz` format. The last step will be to unzip the files. The unzipping and `datalad save` steps will take a few minutes. Fourth tea refill is a charm!\n",
    "\n",
    "The `%%bash` in the cell tells the entire cell to run the code in bash."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 28,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "submission_32739\n"
     ]
    }
   ],
   "source": [
    "! ls /home/jovyan/ABCDndar3/tar_files"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 30,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "create(error): /home/jovyan/ABCDndar3/image_files (dataset) [will not create a dataset in a non-empty directory, use `force` option to ignore]\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[ERROR] will not create a dataset in a non-empty directory, use `force` option to ignore [create(/home/jovyan/ABCDndar3/image_files)] \n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "# let's use datalad to track the unzipped dataset\n",
    "datalad create /home/jovyan/ABCDndar3/image_files\n",
    "\n",
    "# now unzip the files\n",
    "cd /home/jovyan/ABCDndar3/tar_files\n",
    "for sub in submission_*; do\n",
    "    cd $sub\n",
    "    for f in *.tgz; do\n",
    "        tar zxf $f --directory /home/jovyan/ABCDndar3/image_files\n",
    "    done\n",
    "done"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 31,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "add(ok): dataset_description.json (file)\n",
      "add(ok): sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/anat/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_run-01_T1w.json (file)\n",
      "add(ok): sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/anat/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_run-01_T1w.nii (file)\n",
      "add(ok): sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-01_bold.json (file)\n",
      "add(ok): sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-01_bold.nii (file)\n",
      "add(ok): sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-01_motion.tsv (file)\n",
      "add(ok): sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-02_bold.json (file)\n",
      "add(ok): sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-02_bold.nii (file)\n",
      "add(ok): sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-02_motion.tsv (file)\n",
      "add(ok): sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-03_bold.json (file)\n",
      "add(ok): sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-03_bold.nii (file)\n",
      "add(ok): sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-03_motion.tsv (file)\n",
      "add(ok): sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-04_bold.json (file)\n",
      "add(ok): sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-04_bold.nii (file)\n",
      "add(ok): sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-04_motion.tsv (file)\n",
      "add(ok): sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/anat/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_run-01_T1w.json (file)\n",
      "add(ok): sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/anat/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_run-01_T1w.nii (file)\n",
      "add(ok): sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-01_bold.json (file)\n",
      "add(ok): sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-01_bold.nii (file)\n",
      "add(ok): sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-01_motion.tsv (file)\n",
      "add(ok): sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-02_bold.json (file)\n",
      "add(ok): sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-02_bold.nii (file)\n",
      "add(ok): sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-02_motion.tsv (file)\n",
      "add(ok): sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-03_bold.json (file)\n",
      "add(ok): sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-03_bold.nii (file)\n",
      "add(ok): sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-03_motion.tsv (file)\n",
      "add(ok): sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-04_bold.json (file)\n",
      "add(ok): sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-04_bold.nii (file)\n",
      "add(ok): sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-04_motion.tsv (file)\n",
      "add(ok): sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/anat/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_run-01_T1w.json (file)\n",
      "add(ok): sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/anat/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_run-01_T1w.nii (file)\n",
      "add(ok): sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-01_bold.json (file)\n",
      "add(ok): sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-01_bold.nii (file)\n",
      "add(ok): sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-01_motion.tsv (file)\n",
      "add(ok): sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-02_bold.json (file)\n",
      "add(ok): sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-02_bold.nii (file)\n",
      "add(ok): sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-02_motion.tsv (file)\n",
      "add(ok): sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-03_bold.json (file)\n",
      "add(ok): sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-03_bold.nii (file)\n",
      "add(ok): sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-03_motion.tsv (file)\n",
      "add(ok): sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-04_bold.json (file)\n",
      "add(ok): sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-04_bold.nii (file)\n",
      "add(ok): sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-04_motion.tsv (file)\n",
      "add(ok): sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/anat/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_run-01_T1w.json (file)\n",
      "add(ok): sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/anat/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_run-01_T1w.nii (file)\n",
      "add(ok): sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-01_bold.json (file)\n",
      "add(ok): sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-01_bold.nii (file)\n",
      "add(ok): sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-01_motion.tsv (file)\n",
      "add(ok): sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-02_bold.json (file)\n",
      "add(ok): sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-02_bold.nii (file)\n",
      "add(ok): sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-02_motion.tsv (file)\n",
      "add(ok): sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-03_bold.json (file)\n",
      "add(ok): sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-03_bold.nii (file)\n",
      "add(ok): sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-03_motion.tsv (file)\n",
      "add(ok): sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-04_bold.json (file)\n",
      "add(ok): sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-04_bold.nii (file)\n",
      "add(ok): sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-04_motion.tsv (file)\n",
      "add(ok): sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/anat/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_run-01_T1w.json (file)\n",
      "add(ok): sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/anat/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_run-01_T1w.nii (file)\n",
      "add(ok): sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-01_bold.json (file)\n",
      "add(ok): sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-01_bold.nii (file)\n",
      "add(ok): sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-01_motion.tsv (file)\n",
      "add(ok): sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-02_bold.json (file)\n",
      "add(ok): sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-02_bold.nii (file)\n",
      "add(ok): sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-02_motion.tsv (file)\n",
      "add(ok): sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-03_bold.json (file)\n",
      "add(ok): sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-03_bold.nii (file)\n",
      "add(ok): sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-03_motion.tsv (file)\n",
      "add(ok): sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-04_bold.json (file)\n",
      "add(ok): sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-04_bold.nii (file)\n",
      "add(ok): sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-04_motion.tsv (file)\n",
      "save(ok): . (dataset)\n",
      "action summary:\n",
      "  add (ok: 71)\n",
      "  save (ok: 1)\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "# track the changes in datalad\n",
    "cd /home/jovyan/ABCDndar3/image_files\n",
    "datalad save -m 'add unzipped files from NDA' ."
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Let's look at the log to see the changes we've made to this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 32,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "commit 636a705825ce96aff996f716fcbc5e337b4897db\n",
      "Author: Anna <anna.vannucci@columbia.edu>\n",
      "Date:   Fri Mar 5 01:40:03 2021 +0000\n",
      "\n",
      "    add unzipped files from NDA\n",
      "\n",
      "commit a1fedb66c33bbcf339884625c003cea777d0b613\n",
      "Author: Anna <anna.vannucci@columbia.edu>\n",
      "Date:   Fri Mar 5 01:28:21 2021 +0000\n",
      "\n",
      "    [DATALAD] new dataset\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cd /home/jovyan/ABCDndar3/image_files\n",
    "git log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Success!!\n",
    "# 🎉🎉🎉"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "## Downloading the data using the nda-abcd-s3-downloader\n",
    "\n",
    "The [Developmental Cognition and Neuroimaging Lab (DCAN)](https://www.ohsu.edu/school-of-medicine/developmental-cognition-and-neuroimaging-lab) at Oregon Health & Science University has created a handy tool to make downloading easier. In addition, they have uploaded preprocessed derivatives to facilitate quick analysis. More information about the specific contents and preprocessing pipeline can be found at the [Collection 3165 documentation page](https://collection3165.readthedocs.io/en/stable/release_notes/). The procedure for preparing the final download is similar to the procedure above.\n",
    "\n",
    "First you need to create a package and download the list of files contained in Collection 3165. Below are the instructions for using this tool from the [`nda-abcd-s3-downloader` README](https://github.com/DCAN-Labs/nda-abcd-s3-downloader). \n",
    "\n",
    "1. Navigate to the [NDA website](https://nda.nih.gov/general-query.html?q=query=collections%20~and~%20searchTerm=DCAN%20Labs%20ABCD-BIDS%20MRI%20pipeline%20inputs%20and%20derivatives%20~and~%20orderBy=id%20~and~%20orderDirection=Ascending) \n",
    "2. Under \"Get Data\" select \"Data from Labs\"\n",
    "3. Search for \"DCAN Labs ABCD-BIDS MRI pipeline inputs and derivatives\"\n",
    "4. After clicking on the Collection Title select \"Shared Data\"\n",
    "5. Click \"Add to Cart\" at the bottom\n",
    "6. It will take a minute to update the \"Filter Cart\" in the upper right corner, but when that is done select \"Package/Add to Study\"Select \"Create Package\", name your package accordingly, and click \"Create Package\"\n",
    "- IMPORTANT: Make sure \"Include associated data files\" is deselected or it will automatically attempt to download all the data through the NDA's package manager which is unreliable on such a large dataset. That is why we've created this downloader for you.\n",
    "7. Now download the \"Download Manager\" to actually download the package or use the NDA's nda-tools to download the package from the command line. This may take several minutes.\n",
    "8. After the download is complete find the \"datastructure_manifest.txt\" in the downloaded directory. This is the input S3 file that contains AWS S3 links to every input and derivative for all of the selected subjects and you will need to give the path to this file when calling download.py"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    " Hopefully most of the steps to download the manifest sound familiar. Once you have created the Data Package in the NDA, you can proceed.\n",
    " \n",
    " 9. Let's make a directory and download the data package."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 33,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Running NDATools Version 0.2.3\n",
      "Opening log: /home/jovyan/NDAValidationResults/debug_log_20210305T035615.txt\n"
     ]
    }
   ],
   "source": [
    "! mkdir /home/jovyan/ABCDdcan\n",
    "! downloadcmd 1186298 -dp -d /home/jovyan/ABCDdcan # numbers are the data package ID from nda"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 34,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Collection_Documents\t    fmriresults01.txt\t     README.pdf\n",
      "datastructure_manifest.txt  imagingcollection01.txt\n",
      "experiments\t\t    package_info.txt\n"
     ]
    }
   ],
   "source": [
    "! ls /home/jovyan/ABCDdcan"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "10. We should see `datastructure_manifest.txt` in `/home/jovyan/ABCDdcan`. This is the file that contains all of the s3 links for the input and derivative data. As before, we'll need to filter the larger file, but we'll do so in a different way. But first, we need to clone the [nda-abcd-s3-downloader](https://github.com/DCAN-Labs/nda-abcd-s3-downloader)."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 35,
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Cloning into 'nda-abcd-s3-downloader'...\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cd /home/jovyan/ABCDdcan\n",
    "git clone https://github.com/DCAN-Labs/nda-abcd-s3-downloader.git"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "11. The `nda-abcd-s3-downloader` accepts a `data_subsets.txt` file in which we can specify which subsets of the data we want. Let's download some raw (input) data and some derivative data, but keep them separate. You can see the list of data subsets [here](https://github.com/DCAN-Labs/nda-abcd-s3-downloader/blob/master/data_subsets.txt). First, we'll download the inputs."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 36,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "cd /home/jovyan/ABCDdcan\n",
    "# write inputs of interest to inputs.txt\n",
    "echo inputs.anat.T1w >> inputs.txt\n",
    "echo inputs.func.task-rest >> inputs.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "12. Now that you have the data subsets, you'll also need to create a subject subset file. You can use the same subjects as you did for the above exercise. Create a file called `5subjects.txt` that contains the following format. You can `echo` the GUIDs to a text file as we did with the inputs above, or you can create a text file outside of this notebook.\n",
    "\n",
    "```\n",
    "sub-NDARINVXXXXXXXX\n",
    "sub-NDARINVXXXXXXXX\n",
    "sub-NDARINVXXXXXXXX\n",
    "sub-NDARINVXXXXXXXX\n",
    "sub-NDARINVXXXXXXXX\n",
    "```"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 47,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "sub-NDARINVZVW3HKWN\n",
      "sub-NDARINVXEYPED0K\n",
      "sub-NDARINVZ3638RYW\n",
      "sub-NDARINVXFURZ24F\n",
      "sub-NDARINV0PKCP1P3\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cd /home/jovyan/ABCDdcan\n",
    "\n",
    "# create .txt file\n",
    "touch 5subjects.txt \n",
    "\n",
    "# add 5 guids to the .txt file\n",
    "echo \"sub-NDARINVZVW3HKWN\" > 5subjects.txt \n",
    "echo \"sub-NDARINVXEYPED0K\" >> 5subjects.txt\n",
    "echo \"sub-NDARINVZ3638RYW\" >> 5subjects.txt\n",
    "echo \"sub-NDARINVXFURZ24F\" >> 5subjects.txt\n",
    "echo \"sub-NDARINV0PKCP1P3\" >> 5subjects.txt\n",
    "\n",
    "# view contents of .txt file\n",
    "cat 5subjects.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "13. Now we are ready to download the input data! This step will take a few minutes. You're probably out of tea at this point.\n",
    "\n",
    "#### **NOTE**: The first time you run `download.py`, you'll need to run it from your terminal because it will ask for your NDA credentials"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 48,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "create(error): /home/jovyan/ABCDdcan/image_files (dataset) [will not create a dataset in a non-empty directory, use `force` option to ignore]\n",
      "Derivatives downloader called at 2021:03:05 04:17 with:\n",
      "\n",
      "Enter your NIMH Data Archives username: "
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[ERROR] will not create a dataset in a non-empty directory, use `force` option to ignore [create(/home/jovyan/ABCDdcan/image_files)] \n",
      "Traceback (most recent call last):\n",
      "  File \"nda-abcd-s3-downloader/download.py\", line 386, in <module>\n",
      "    _cli()\n",
      "  File \"nda-abcd-s3-downloader/download.py\", line 109, in _cli\n",
      "    make_nda_token(args.credentials)\n",
      "  File \"nda-abcd-s3-downloader/download.py\", line 304, in make_nda_token\n",
      "    username = input(\"\\nEnter your NIMH Data Archives username: \")\n",
      "EOFError: EOF when reading a line\n"
     ]
    },
    {
     "ename": "CalledProcessError",
     "evalue": "Command 'b\"\\n# let's use datalad to track the unzipped dataset\\ndatalad create /home/jovyan/ABCDdcan/image_files\\n\\n## NOTE: the first time you run download.py, you'll need to run it from your terminal because it will ask for your NDA credentials\\ncd /home/jovyan/ABCDdcan/\\n\\n# run the downloader\\nnda-abcd-s3-downloader/download.py -i datastructure_manifest.txt -o image_files -s 5subjects.txt -d inputs.txt\\n\"' returned non-zero exit status 1.",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mCalledProcessError\u001b[0m                        Traceback (most recent call last)",
      "\u001b[0;32m<ipython-input-48-1aa7b95a7a59>\u001b[0m in \u001b[0;36m<module>\u001b[0;34m\u001b[0m\n\u001b[0;32m----> 1\u001b[0;31m \u001b[0mget_ipython\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mrun_cell_magic\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m'bash'\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m''\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m\"\\n# let's use datalad to track the unzipped dataset\\ndatalad create /home/jovyan/ABCDdcan/image_files\\n\\n## NOTE: the first time you run download.py, you'll need to run it from your terminal because it will ask for your NDA credentials\\ncd /home/jovyan/ABCDdcan/\\n\\n# run the downloader\\nnda-abcd-s3-downloader/download.py -i datastructure_manifest.txt -o image_files -s 5subjects.txt -d inputs.txt\\n\"\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m",
      "\u001b[0;32m/opt/conda/lib/python3.8/site-packages/IPython/core/interactiveshell.py\u001b[0m in \u001b[0;36mrun_cell_magic\u001b[0;34m(self, magic_name, line, cell)\u001b[0m\n\u001b[1;32m   2380\u001b[0m             \u001b[0;32mwith\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mbuiltin_trap\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2381\u001b[0m                 \u001b[0margs\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;34m(\u001b[0m\u001b[0mmagic_arg_s\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m-> 2382\u001b[0;31m                 \u001b[0mresult\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mfn\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0margs\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mkwargs\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m   2383\u001b[0m             \u001b[0;32mreturn\u001b[0m \u001b[0mresult\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m   2384\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.8/site-packages/IPython/core/magics/script.py\u001b[0m in \u001b[0;36mnamed_script_magic\u001b[0;34m(line, cell)\u001b[0m\n\u001b[1;32m    140\u001b[0m             \u001b[0;32melse\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    141\u001b[0m                 \u001b[0mline\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0mscript\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 142\u001b[0;31m             \u001b[0;32mreturn\u001b[0m \u001b[0mself\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mshebang\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mline\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    143\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    144\u001b[0m         \u001b[0;31m# write a basic docstring:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m<decorator-gen-103>\u001b[0m in \u001b[0;36mshebang\u001b[0;34m(self, line, cell)\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.8/site-packages/IPython/core/magic.py\u001b[0m in \u001b[0;36m<lambda>\u001b[0;34m(f, *a, **k)\u001b[0m\n\u001b[1;32m    185\u001b[0m     \u001b[0;31m# but it's overkill for just that one bit of state.\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    186\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0mmagic_deco\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 187\u001b[0;31m         \u001b[0mcall\u001b[0m \u001b[0;34m=\u001b[0m \u001b[0;32mlambda\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m:\u001b[0m \u001b[0mf\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m*\u001b[0m\u001b[0ma\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0;34m**\u001b[0m\u001b[0mk\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    188\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    189\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0mcallable\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0marg\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;32m/opt/conda/lib/python3.8/site-packages/IPython/core/magics/script.py\u001b[0m in \u001b[0;36mshebang\u001b[0;34m(self, line, cell)\u001b[0m\n\u001b[1;32m    243\u001b[0m             \u001b[0msys\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mstderr\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mflush\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    244\u001b[0m         \u001b[0;32mif\u001b[0m \u001b[0margs\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mraise_error\u001b[0m \u001b[0;32mand\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreturncode\u001b[0m\u001b[0;34m!=\u001b[0m\u001b[0;36m0\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0;32m--> 245\u001b[0;31m             \u001b[0;32mraise\u001b[0m \u001b[0mCalledProcessError\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mp\u001b[0m\u001b[0;34m.\u001b[0m\u001b[0mreturncode\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0moutput\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0mout\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mstderr\u001b[0m\u001b[0;34m=\u001b[0m\u001b[0merr\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n\u001b[0m\u001b[1;32m    246\u001b[0m \u001b[0;34m\u001b[0m\u001b[0m\n\u001b[1;32m    247\u001b[0m     \u001b[0;32mdef\u001b[0m \u001b[0m_run_script\u001b[0m\u001b[0;34m(\u001b[0m\u001b[0mself\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mp\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mcell\u001b[0m\u001b[0;34m,\u001b[0m \u001b[0mto_close\u001b[0m\u001b[0;34m)\u001b[0m\u001b[0;34m:\u001b[0m\u001b[0;34m\u001b[0m\u001b[0;34m\u001b[0m\u001b[0m\n",
      "\u001b[0;31mCalledProcessError\u001b[0m: Command 'b\"\\n# let's use datalad to track the unzipped dataset\\ndatalad create /home/jovyan/ABCDdcan/image_files\\n\\n## NOTE: the first time you run download.py, you'll need to run it from your terminal because it will ask for your NDA credentials\\ncd /home/jovyan/ABCDdcan/\\n\\n# run the downloader\\nnda-abcd-s3-downloader/download.py -i datastructure_manifest.txt -o image_files -s 5subjects.txt -d inputs.txt\\n\"' returned non-zero exit status 1."
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "# let's use datalad to track the unzipped dataset\n",
    "datalad create /home/jovyan/ABCDdcan/image_files\n",
    "\n",
    "## NOTE: the first time you run download.py, you'll need to run it from your terminal because it will ask for your NDA credentials\n",
    "cd /home/jovyan/ABCDdcan/\n",
    "\n",
    "# run the downloader\n",
    "nda-abcd-s3-downloader/download.py -i datastructure_manifest.txt -o image_files -s 5subjects.txt -d inputs.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "14. Let's check to see if the input data we downloaded are in proper BIDS format. To do this we'll use the [bids-validator](https://github.com/bids-standard/bids-validator). We'll build a singularity container from the docker image so that we can run the validator on the JupyterHub."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "2021/03/05 19:08:03 bufio.Scanner: token too long\n",
      "Build target 'bids_validator-1.6.1.simg' already exists and will be deleted during the build process. Do you want to continue? [N/y]^C\n",
      "\u001b[31mFATAL:  \u001b[0m While checking build target: stopping build\n"
     ]
    }
   ],
   "source": [
    "! singularity build bids_validator-1.6.1.simg docker://bids/validator:v1.6.1"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Now let's see if the input data are in BIDS."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "bids-validator@1.6.1\n",
      "\n",
      "\t\u001b[33m1: [WARN] You should define 'SliceTiming' for this file. If you don't provide this information slice time correction will not be possible. 'Slice Timing' is the time at which each slice was acquired within each volume (frame) of the acquisition. Slice timing is not slice order -- rather, it is a list of times containing the time (in seconds) of each slice acquisition in relation to the beginning of volume acquisition. (code: 13 - SLICE_TIMING_NOT_DEFINED)\u001b[39m\n",
      "\t\t./sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-01_bold.nii.gz\n",
      "\t\t./sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-02_bold.nii.gz\n",
      "\t\t./sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-03_bold.nii.gz\n",
      "\t\t./sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-04_bold.nii.gz\n",
      "\t\t./sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-01_bold.nii.gz\n",
      "\t\t./sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-02_bold.nii.gz\n",
      "\t\t./sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-03_bold.nii.gz\n",
      "\t\t./sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-04_bold.nii.gz\n",
      "\t\t./sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-01_bold.nii.gz\n",
      "\t\t./sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-02_bold.nii.gz\n",
      "\t\t\u001b[33m... and 10 more files having this issue (Use --verbose to see them all).\u001b[39m\n",
      "\n",
      "\u001b[36m\tPlease visit https://neurostars.org/search?q=SLICE_TIMING_NOT_DEFINED for existing conversations about this issue.\u001b[39m\n",
      "\n",
      "\t\u001b[33m2: [WARN] Not all subjects contain the same files. Each subject should contain the same number of files with the same naming unless some files are known to be missing. (code: 38 - INCONSISTENT_SUBJECTS)\u001b[39m\n",
      "\t\t./sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/anat/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_T1w.json\n",
      "\t\t\tEvidence: Subject: sub-NDARINV0PKCP1P3; Missing file: sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_T1w.json\n",
      "\t\t./sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/anat/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_T1w.nii.gz\n",
      "\t\t\tEvidence: Subject: sub-NDARINV0PKCP1P3; Missing file: sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_T1w.nii.gz\n",
      "\t\t./sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/anat/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_rec-normalized_T1w.json\n",
      "\t\t\tEvidence: Subject: sub-NDARINV0PKCP1P3; Missing file: sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_rec-normalized_T1w.json\n",
      "\t\t./sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/anat/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_rec-normalized_T1w.nii.gz\n",
      "\t\t\tEvidence: Subject: sub-NDARINV0PKCP1P3; Missing file: sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_rec-normalized_T1w.nii.gz\n",
      "\t\t./sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/anat/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_T1w.json\n",
      "\t\t\tEvidence: Subject: sub-NDARINVXEYPED0K; Missing file: sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_T1w.json\n",
      "\t\t./sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/anat/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_T1w.nii.gz\n",
      "\t\t\tEvidence: Subject: sub-NDARINVXEYPED0K; Missing file: sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_T1w.nii.gz\n",
      "\t\t./sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/anat/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_rec-normalized_run-01_T1w.json\n",
      "\t\t\tEvidence: Subject: sub-NDARINVXEYPED0K; Missing file: sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_rec-normalized_run-01_T1w.json\n",
      "\t\t./sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/anat/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_rec-normalized_run-01_T1w.nii.gz\n",
      "\t\t\tEvidence: Subject: sub-NDARINVXEYPED0K; Missing file: sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_rec-normalized_run-01_T1w.nii.gz\n",
      "\t\t./sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/anat/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_rec-normalized_run-02_T1w.json\n",
      "\t\t\tEvidence: Subject: sub-NDARINVXEYPED0K; Missing file: sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_rec-normalized_run-02_T1w.json\n",
      "\t\t./sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/anat/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_rec-normalized_run-02_T1w.nii.gz\n",
      "\t\t\tEvidence: Subject: sub-NDARINVXEYPED0K; Missing file: sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_rec-normalized_run-02_T1w.nii.gz\n",
      "\t\t\u001b[33m... and 18 more files having this issue (Use --verbose to see them all).\u001b[39m\n",
      "\n",
      "\u001b[36m\tPlease visit https://neurostars.org/search?q=INCONSISTENT_SUBJECTS for existing conversations about this issue.\u001b[39m\n",
      "\n",
      "\t\u001b[33m3: [WARN] Not all subjects/sessions/runs have the same scanning parameters. (code: 39 - INCONSISTENT_PARAMETERS)\u001b[39m\n",
      "\t\t./sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-01_bold.nii.gz\n",
      "\t\t./sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-02_bold.nii.gz\n",
      "\t\t./sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-03_bold.nii.gz\n",
      "\t\t./sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-04_bold.nii.gz\n",
      "\t\t./sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/anat/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_T1w.nii.gz\n",
      "\t\t./sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-01_bold.nii.gz\n",
      "\t\t./sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-02_bold.nii.gz\n",
      "\t\t./sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-03_bold.nii.gz\n",
      "\t\t./sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-04_bold.nii.gz\n",
      "\n",
      "\u001b[36m\tPlease visit https://neurostars.org/search?q=INCONSISTENT_PARAMETERS for existing conversations about this issue.\u001b[39m\n",
      "\n",
      "\n",
      "        \u001b[34m\u001b[4mSummary:\u001b[24m\u001b[39m                \u001b[34m\u001b[4mAvailable Tasks:\u001b[24m\u001b[39m                 \u001b[34m\u001b[4mAvailable Modalities:\u001b[24m\u001b[39m \n",
      "        56 Files, 3.72GB        Resting State (eyes open)        T1w                   \n",
      "        5 - Subjects                                             bold                  \n",
      "        1 - Session                                                                    \n",
      "\n",
      "\n",
      "\u001b[36m\tIf you have any questions, please post on https://neurostars.org/tags/bids.\u001b[39m\n",
      "\n"
     ]
    }
   ],
   "source": [
    "! singularity run /home/jovyan/bids_validator-1.6.1.simg /home/jovyan/ABCDdcan/image_files"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "It looks like the data are in BIDS! (Warnings are ok, but something you should be aware of)"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "15. Now let's download some derivatives! You can use the same subjects file."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "metadata": {},
   "outputs": [],
   "source": [
    "%%bash\n",
    "\n",
    "cd /home/jovyan/ABCDdcan\n",
    "\n",
    "# make subsets file\n",
    "cat ./nda-abcd-s3-downloader/data_subsets.txt | grep derivatives | grep rest >> derivatives.txt\n",
    "cat ./nda-abcd-s3-downloader/data_subsets.txt | grep derivatives | grep T1w >> derivatives.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "16. Now download the derivatives. `download.py` automatically adds the derivatives directory. So we'll have to force a subdataset with datalad after the download."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Process is terminated.\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cd /home/jovyan/ABCDdcan/\n",
    "\n",
    "# run the downloader\n",
    "nda-abcd-s3-downloader/download.py -i datastructure_manifest.txt -o image_files/derivatives -s 5subjects.txt -d derivatives.txt"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Downloaded data using commands in terminal to monitor progress more easily.\n",
    "\n",
    "Sample of the s3:// file path for each participant: \n",
    "`s3://NDAR_Central_2/submission_23337/derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_bold_atlas-Gordon2014FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii`"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "***\n",
    "\n",
    "18. Now use datalad to create a subdataset and save the subdataset's state."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 8,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "create(ok): . (dataset)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/anat/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_hemi-L_space-T1w_mesh-native_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/anat/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_hemi-R_space-T1w_mesh-fsLR32k_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/anat/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_hemi-R_space-T1w_mesh-native_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_bold_atlas-Gordon2014FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_bold_atlas-HCP2016FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_bold_atlas-Markov2012FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_bold_atlas-Power2011FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_bold_atlas-Yeo2011FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_bold_desc-filtered_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_desc-filtered_motion_mask.mat (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_desc-filteredwithoutliers_motion_mask.mat (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-1_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-1_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-1_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-1_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-1_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-1_space-MNI_bold.nii.gz (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-2_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-2_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-2_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-2_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-2_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-2_space-MNI_bold.nii.gz (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-3_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-3_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-3_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-3_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-3_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-4_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-4_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-4_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-4_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINV0PKCP1P3/ses-baselineYear1Arm1/func/sub-NDARINV0PKCP1P3_ses-baselineYear1Arm1_task-rest_run-4_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/anat/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_T1w_space-MNI_brain.nii.gz (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/anat/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_T1w_space-MNI_head.nii.gz (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/anat/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_hemi-L_space-T1w_mesh-fsLR32k_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/anat/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_hemi-L_space-T1w_mesh-native_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/anat/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_hemi-R_space-T1w_mesh-fsLR32k_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/anat/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_hemi-R_space-T1w_mesh-native_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_bold_atlas-Gordon2014FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_bold_atlas-HCP2016FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_bold_atlas-Markov2012FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_bold_atlas-Power2011FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_bold_atlas-Yeo2011FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_bold_desc-filtered_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_desc-filtered_motion_mask.mat (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_desc-filteredwithoutliers_motion_mask.mat (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-1_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-1_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-1_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-1_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-1_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-1_space-MNI_bold.nii.gz (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-2_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-2_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-2_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-2_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-2_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-2_space-MNI_bold.nii.gz (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-3_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-3_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-3_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-3_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-3_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-4_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-4_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-4_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-4_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXEYPED0K/ses-baselineYear1Arm1/func/sub-NDARINVXEYPED0K_ses-baselineYear1Arm1_task-rest_run-4_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/anat/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_T1w_space-MNI_brain.nii.gz (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/anat/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_T1w_space-MNI_head.nii.gz (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/anat/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_hemi-L_space-T1w_mesh-fsLR32k_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/anat/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_hemi-L_space-T1w_mesh-native_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/anat/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_hemi-R_space-T1w_mesh-fsLR32k_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/anat/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_hemi-R_space-T1w_mesh-native_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_bold_atlas-Gordon2014FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_bold_atlas-HCP2016FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_bold_atlas-Markov2012FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_bold_atlas-Power2011FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_bold_atlas-Yeo2011FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_bold_desc-filtered_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_desc-filtered_motion_mask.mat (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_desc-filteredwithoutliers_motion_mask.mat (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-1_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-1_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-1_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-1_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-1_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-1_space-MNI_bold.nii.gz (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-2_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-2_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-2_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-2_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-2_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-2_space-MNI_bold.nii.gz (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-3_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-3_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-3_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-3_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-3_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-4_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-4_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-4_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-4_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVXFURZ24F/ses-baselineYear1Arm1/func/sub-NDARINVXFURZ24F_ses-baselineYear1Arm1_task-rest_run-4_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/anat/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_T1w_space-MNI_brain.nii.gz (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/anat/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_T1w_space-MNI_head.nii.gz (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/anat/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_hemi-L_space-T1w_mesh-fsLR32k_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/anat/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_hemi-L_space-T1w_mesh-native_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/anat/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_hemi-R_space-T1w_mesh-fsLR32k_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/anat/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_hemi-R_space-T1w_mesh-native_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_bold_atlas-Gordon2014FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_bold_atlas-HCP2016FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_bold_atlas-Markov2012FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_bold_atlas-Power2011FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_bold_atlas-Yeo2011FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_bold_desc-filtered_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_desc-filtered_motion_mask.mat (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_desc-filteredwithoutliers_motion_mask.mat (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-1_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-1_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-1_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-1_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-1_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-1_space-MNI_bold.nii.gz (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-2_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-2_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-2_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-2_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-2_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-2_space-MNI_bold.nii.gz (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-3_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-3_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-3_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-3_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-3_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-4_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-4_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-4_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-4_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZ3638RYW/ses-baselineYear1Arm1/func/sub-NDARINVZ3638RYW_ses-baselineYear1Arm1_task-rest_run-4_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/anat/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_T1w_space-MNI_brain.nii.gz (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/anat/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_T1w_space-MNI_head.nii.gz (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/anat/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_hemi-L_space-T1w_mesh-fsLR32k_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/anat/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_hemi-L_space-T1w_mesh-native_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/anat/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_hemi-R_space-T1w_mesh-fsLR32k_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/anat/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_hemi-R_space-T1w_mesh-native_midthickness.surf.gii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_bold_atlas-Gordon2014FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_bold_atlas-HCP2016FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_bold_atlas-Markov2012FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_bold_atlas-Power2011FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_bold_atlas-Yeo2011FreeSurferSubcortical_desc-filtered_timeseries.ptseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_bold_desc-filtered_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_desc-filtered_motion_mask.mat (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_desc-filteredwithoutliers_motion_mask.mat (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-1_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-1_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-1_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-1_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-1_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-1_space-MNI_bold.nii.gz (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-2_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-2_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-2_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-2_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-2_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-2_space-MNI_bold.nii.gz (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-3_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-3_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-3_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-3_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-3_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-4_bold_timeseries.dtseries.nii (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-4_desc-filtered_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-4_desc-filteredincludingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-4_desc-includingFD_motion.tsv (file)\n",
      "add(ok): derivatives/abcd-hcp-pipeline/sub-NDARINVZVW3HKWN/ses-baselineYear1Arm1/func/sub-NDARINVZVW3HKWN_ses-baselineYear1Arm1_task-rest_run-4_motion.tsv (file)\n",
      "save(ok): . (dataset)\n",
      "action summary:\n",
      "  add (ok: 177)\n",
      "  save (ok: 1)\n"
     ]
    },
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "[INFO] Creating a new annex repo at /home/jovyan/ABCDdcan/image_files/derivatives \n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cd /home/jovyan/ABCDdcan/image_files\n",
    "# force the creation of a subdataset\n",
    "datalad create -d derivatives --force\n",
    "\n",
    "# now save the subdataset\n",
    "cd derivatives\n",
    "datalad save -m 'add T1w and rest derivatives'"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "Finally, we can look at the log to see the changes we've made to this dataset."
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 9,
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "commit 6f0d548efaa9a45d0f9087cfddaed8f1ffd3e305\n",
      "Author: Anna <anna.vannucci@columbia.edu>\n",
      "Date:   Fri Mar 5 04:11:37 2021 +0000\n",
      "\n",
      "    [DATALAD] new dataset\n"
     ]
    }
   ],
   "source": [
    "%%bash\n",
    "\n",
    "cd /home/jovyan/ABCDdcan/image_files\n",
    "git log"
   ]
  },
  {
   "cell_type": "markdown",
   "metadata": {},
   "source": [
    "### Success!!\n",
    "# 🎉🎉🎉\n",
    "\n",
    "You should probably buy more tea... ☕"
   ]
  }
 ],
 "metadata": {
  "kernelspec": {
   "display_name": "Python 3",
   "language": "python",
   "name": "python3"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.8.6"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 4
}