Data Querying Tool

HOME > SETUP > DATA QUERY TOOL

The Data Query Tool is an open-source extension of LORIS which enables users to query and download datasets. Front-end users can design and store queries to extract any data available in LORIS, including imaging data.

Note: SSL is not fully supported for the Data Querying Tool. Individual Loris instances should set up their own SSL.

Installing the Data Query Tool

Quick setup:

• Ubuntu: apt-get install couchdb (needs sudo)
• in CouchDB: set up an admin credential and create a new database
• replicate our public couchbase using curl - see Data Query Tool Readme
• run the data loading scripts (below)

Please see the Data Query Tool Readme (in beta) for more complete installation instructions. Detailed instructions for some systems are available in the DQT repository wiki

Loading the Data Query Tool

Custom php scripts are provided in the LORIS codebase for loading data into the Data Query Tool.

First, generate the Data Dictionary by creating a special file ip_output.txt of all instrument fields, and then loading them into the parameter_type table:

1. For all PHP instruments: run the lorisform_parser.php script to generate ip_output.txt

cd /var/www/loris/tools
find ../project/instruments/NDB_BVL_Instrument_*.class.inc | php lorisform_parser.php 

2. Append (do not overwrite) to ip_output.txt all fields from all linst files:

cat /var/www/loris/project/instruments/*.linst >> /var/www/loris/tools/ip_output.txt

3. Then run the tools/exporters/data_dictionary_builder.php script -- but before you do:

• Ensure the parameter_type_category table contains a record where Type = 'Instrument'
• Back up your parameter_type table, since the script will delete and re-load all Instrument type fields Then run the script from the tools directory:

php data_dictionary_builder.php 

Second, to load data into the DQT -- In your LORIS tools/ directory, run the CouchDB_Import_* scripts to import each type of data :

• Base candidate data : CouchDB_Import_Demographics.php : Run first to import base candidate data
• Instrument data : CouchDB_Import_Instruments.php - omit if Behavioural data not used
• Imaging (MRI) data : CouchDB_Import_MRI.php - omit if Imaging data not used
• Radiological review module data : CouchDB_Import_RadiologicalReview.php - omit unless module in use

Note that the path is assumed to be var/www/loris however your own path may be var/www/<project-name>, depending on your setup.

cd /var/www/loris/tools
php CouchDB_Import_Demographics.php
php CouchDB_Import_Instruments.php
php CouchDB_Import_MRI.php
php CouchDB_Import_RadiologicalReview.php

CouchDB_Import_*.php scripts are provided in the tools/ directory to push Demographic, Instrument, MRI and Radiological Review data from LORIS into the DQT. Note that CouchDB_Import_Demographics.php must be run first.

Upon initial import of data, it may take between a few minutes and a few hours to build the initial indexes, depending on the size of your dataset.

Imaging files in the Data Query Tool

The MRI import script may require some editing or configuration to find the exact path for all imaging data. Note that only QC'd ("Selected") minc files (i.e. Selected as the best acquisition for a given scan type e.g. t1) for valid timepoints are automatically loaded in the DQT by this script.

MRI file data is loaded in the DQT under the category mri_data, fieldname "Selected_$SCANTYPE" e.g. Selected_t1. Imaging files should then be available for download under the "View Data" tab either by clicking the link after running the query, or clicking on "Download all as Zip" button.

Downloading data

In the View Data tab, click the "Run Query" button. Then scroll down and click "Download as CSV" to retrieve the data table of results you see on the screen.

Note that the "Download all as Zip" button works only if Imaging files have been loaded and selected.

Troubleshooting

• If queries do not run immediately after setting up the DQT, note that it may take between a few minutes and a few hours to build the initial indexes, depending on the size of your dataset
• Futon, the built-in administrative interface for CouchDB, is highly useful for debugging

User Help

A User Guide is available within the DQT's Help section, by clicking on Question Mark icon in top menu bar.