From 17747567cd35c44b0adf8adb57d9a5ae00383c38 Mon Sep 17 00:00:00 2001 From: MounaSafiHarab Date: Fri, 5 Jan 2018 13:18:48 -0500 Subject: [PATCH 01/12] Add SPEC/README for Imaging Uploader --- modules/imaging_uploader/README.md | 97 ++++++++++++++++++++++++++++-- 1 file changed, 93 insertions(+), 4 deletions(-) diff --git a/modules/imaging_uploader/README.md b/modules/imaging_uploader/README.md index 3f66e169733..6b43a89ce24 100644 --- a/modules/imaging_uploader/README.md +++ b/modules/imaging_uploader/README.md @@ -1,9 +1,21 @@ -To enable the Imaging Uploader to handle large files, please update the apache configuration file -with the following values. +# Imaging Uploader -File to update : `/etc/php5/apache2/php.ini` +## Purpose -Sample values: +The imaging uploader is intended to allow users to upload, browse, and track +pipeline insertion progress. + + +## Requirements + +A successful scan upload should obey the following requirements: + + +### Install Configurations + +To enable the Imaging Uploader to handle large files, please update the apache +configuration file (located in `/etc/php/7.0/apache2/php.ini` for PHP 7.0) with +the following sample values: ``` session.gc_maxlifetime = 10800 @@ -12,3 +24,80 @@ max_execution_time = 10800 upload_max_filesize = 1024M post_max_size = 1024M ``` + +### Uploaded file + +The uploaded file is expected to be of one of the following types: +`.tgz, .tar.gz or .zip`. The filename should follow the: +`PSCID_CandID_VisitLabel_*OptionalSuffix*.zip` naming convention for a `.zip` +file. It is also expected that the candidate and visit are already created in +the database. + + +## Intended Users + +The three primary types of users are: +1. Imaging specialists using the modules to do online QC +2. Project radiologists viewing images to report incidental findings +3. Site coordinators or researchers ensuring their uploaded scans have + been processed and inserted into LORIS. + +## Scope + +The imaging uploader has the following built-in capabilities to facilitate +timely scan insertion into the LORIS database: + +1. Upload imaging scans using the `Upload` that can subsequently be inserted +into the database automatically using a collection of perl scripts provided in +[LORIS-MRI](https://github.com/aces/Loris-MRI). The launching of the insertion +can also be achieved manually +2. Browse uploaded scans using the `Browse` tab, which displays information +about who uploaded the scan, when it was uploaded, and an `UploadID` and +`UploadLocation` that can be used to trigger the LORIS-MRI insertion manually +3. Track scan insertion status through the `Progress` column and a `Log Viewer` +which displays in either `Summary` or `Detailed` mode relevant feedback messages +about insertion success or failure causes +4. Uploaded scans can be overwritten if insertion is not started or `Failed` +(i.e. if the `Progress` is neither `In progres` nor `Success`. + + +NOT in scope: + +The imaging uploader does NOT read the DICOM files within the uploaded scans. +As such, it does not check if the files within the uploaded scan are of the +expected DICOM type, nor whether the PatientName and/or PatientID DICOM headers +are properly de-identified according to the LORIS convention. This check is +however done as the first step on the LORIS-MRi side; i.e. as soon as the +insertion pipeline is triggered. + +## Permissions + +The imaging uploader module uses one permission called `imaging_uploader` that +is necessary to have access to the module and gives the user the ability to +upload and browse all scans uploaded to the database. + + +## Database Configurations + +The imaging uploader has the following configurations that affect its usage + +ImagingUploaderAutoLaunch - This setting determines whether the insertion + pipeline that archives the images and is triggered automatically or + manually. + +MRIUploadIncomingPath - This setting determines where on the filesystem the + uploader is to place the uploaded file. Default location is + `/data/incoming/`. This directory is created during the installation of + LORIS-MRI. + +patientNameRegex - Unused. + + +## Interactions with LORIS + +- Links to other LORIS modules archived DICOMs acessible from the hyperlink in +the `TarchiveInfo` column +- MINC images are accessible through the hyperlink in the +`Number of MincInserted` column +- If present, MRI violated scans (as defined by the study MRI protocol) can be +accessed through the hyperlink in the `Number of MincCreated` column. From 5ca67c938d19c1c01ddc69af5a8d05fe9b66a4df Mon Sep 17 00:00:00 2001 From: MounaSafiHarab Date: Fri, 5 Jan 2018 13:20:50 -0500 Subject: [PATCH 02/12] intended users --- modules/imaging_uploader/README.md | 7 ++----- 1 file changed, 2 insertions(+), 5 deletions(-) diff --git a/modules/imaging_uploader/README.md b/modules/imaging_uploader/README.md index 6b43a89ce24..1065ac563c7 100644 --- a/modules/imaging_uploader/README.md +++ b/modules/imaging_uploader/README.md @@ -36,11 +36,8 @@ the database. ## Intended Users -The three primary types of users are: -1. Imaging specialists using the modules to do online QC -2. Project radiologists viewing images to report incidental findings -3. Site coordinators or researchers ensuring their uploaded scans have - been processed and inserted into LORIS. +The three primary types of users are MRI technicians or site coordinators to +upload imaging scans for registered LORIS candidates and timepoint. ## Scope From c2fcf155d71d8ad88351a06abf01dff00cae8da6 Mon Sep 17 00:00:00 2001 From: MounaSafiHarab Date: Fri, 5 Jan 2018 13:21:03 -0500 Subject: [PATCH 03/12] typo --- modules/imaging_uploader/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/imaging_uploader/README.md b/modules/imaging_uploader/README.md index 1065ac563c7..0e327efd22c 100644 --- a/modules/imaging_uploader/README.md +++ b/modules/imaging_uploader/README.md @@ -37,7 +37,7 @@ the database. ## Intended Users The three primary types of users are MRI technicians or site coordinators to -upload imaging scans for registered LORIS candidates and timepoint. +upload imaging scans for registered LORIS candidates and timepoints. ## Scope From 5ab2f1b4f9bdaefca912b27835139261eb9692ad Mon Sep 17 00:00:00 2001 From: MounaSafiHarab Date: Fri, 5 Jan 2018 13:30:55 -0500 Subject: [PATCH 04/12] typo2 --- modules/imaging_uploader/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/modules/imaging_uploader/README.md b/modules/imaging_uploader/README.md index 0e327efd22c..c9eb38c6cf4 100644 --- a/modules/imaging_uploader/README.md +++ b/modules/imaging_uploader/README.md @@ -36,7 +36,7 @@ the database. ## Intended Users -The three primary types of users are MRI technicians or site coordinators to +The primary users are MRI technicians or site coordinators to upload imaging scans for registered LORIS candidates and timepoints. ## Scope @@ -92,9 +92,9 @@ patientNameRegex - Unused. ## Interactions with LORIS -- Links to other LORIS modules archived DICOMs acessible from the hyperlink in +1. Links to other LORIS modules archived DICOMs acessible from the hyperlink in the `TarchiveInfo` column -- MINC images are accessible through the hyperlink in the +2. MINC images are accessible through the hyperlink in the `Number of MincInserted` column -- If present, MRI violated scans (as defined by the study MRI protocol) can be +3. If present, MRI violated scans (as defined by the study MRI protocol) can be accessed through the hyperlink in the `Number of MincCreated` column. From 60e445528eb556dc781b0ef5427a5978cfec97c9 Mon Sep 17 00:00:00 2001 From: MounaSafiHarab Date: Fri, 5 Jan 2018 13:32:48 -0500 Subject: [PATCH 05/12] typo3 --- modules/imaging_uploader/README.md | 8 ++++---- 1 file changed, 4 insertions(+), 4 deletions(-) diff --git a/modules/imaging_uploader/README.md b/modules/imaging_uploader/README.md index c9eb38c6cf4..987ee76e076 100644 --- a/modules/imaging_uploader/README.md +++ b/modules/imaging_uploader/README.md @@ -44,7 +44,7 @@ upload imaging scans for registered LORIS candidates and timepoints. The imaging uploader has the following built-in capabilities to facilitate timely scan insertion into the LORIS database: -1. Upload imaging scans using the `Upload` that can subsequently be inserted +1. Upload imaging scans using the `Upload` tab that can subsequently be inserted into the database automatically using a collection of perl scripts provided in [LORIS-MRI](https://github.com/aces/Loris-MRI). The launching of the insertion can also be achieved manually @@ -54,8 +54,8 @@ about who uploaded the scan, when it was uploaded, and an `UploadID` and 3. Track scan insertion status through the `Progress` column and a `Log Viewer` which displays in either `Summary` or `Detailed` mode relevant feedback messages about insertion success or failure causes -4. Uploaded scans can be overwritten if insertion is not started or `Failed` -(i.e. if the `Progress` is neither `In progres` nor `Success`. +4. Uploaded scans can be overwritten if insertion is `Not Started` or `Failed` +(i.e. if the `Progress` is neither `In Progres` nor `Success`. NOT in scope: @@ -76,7 +76,7 @@ upload and browse all scans uploaded to the database. ## Database Configurations -The imaging uploader has the following configurations that affect its usage +The imaging uploader has the following configurations that affect its usage: ImagingUploaderAutoLaunch - This setting determines whether the insertion pipeline that archives the images and is triggered automatically or From e93e9f9492fa84db0d9dc02d90ae02d29de63d2b Mon Sep 17 00:00:00 2001 From: MounaSafiHarab Date: Fri, 5 Jan 2018 13:52:00 -0500 Subject: [PATCH 06/12] some of Daves feedback --- modules/imaging_uploader/README.md | 53 ++++++++++++++---------------- 1 file changed, 24 insertions(+), 29 deletions(-) diff --git a/modules/imaging_uploader/README.md b/modules/imaging_uploader/README.md index 987ee76e076..d8e4165f11c 100644 --- a/modules/imaging_uploader/README.md +++ b/modules/imaging_uploader/README.md @@ -8,30 +8,13 @@ pipeline insertion progress. ## Requirements -A successful scan upload should obey the following requirements: - - -### Install Configurations - -To enable the Imaging Uploader to handle large files, please update the apache -configuration file (located in `/etc/php/7.0/apache2/php.ini` for PHP 7.0) with -the following sample values: - -``` -session.gc_maxlifetime = 10800 -max_input_time = 10800 -max_execution_time = 10800 -upload_max_filesize = 1024M -post_max_size = 1024M -``` - -### Uploaded file - -The uploaded file is expected to be of one of the following types: -`.tgz, .tar.gz or .zip`. The filename should follow the: -`PSCID_CandID_VisitLabel_*OptionalSuffix*.zip` naming convention for a `.zip` -file. It is also expected that the candidate and visit are already created in -the database. +For a successful upload: +1. The uploaded file is expected to be of one of the following types: +`.tgz, .tar.gz or .zip`. +2. The filename should follow the: +`PSCID_CandID_VisitLabel_*OptionalSuffix*` naming convention. +3. It is expected that the candidate and visit are already created in the +database. ## Intended Users @@ -64,7 +47,7 @@ The imaging uploader does NOT read the DICOM files within the uploaded scans. As such, it does not check if the files within the uploaded scan are of the expected DICOM type, nor whether the PatientName and/or PatientID DICOM headers are properly de-identified according to the LORIS convention. This check is -however done as the first step on the LORIS-MRi side; i.e. as soon as the +however done as the first step on the LORIS-MRI side; i.e. as soon as the insertion pipeline is triggered. ## Permissions @@ -74,10 +57,24 @@ is necessary to have access to the module and gives the user the ability to upload and browse all scans uploaded to the database. -## Database Configurations - +## Configurations The imaging uploader has the following configurations that affect its usage: +#### Install Configurations + +To enable the Imaging Uploader to handle large files, please update the +`php.ini` apache configuration file with the following sample values: + +``` +session.gc_maxlifetime = 10800 +max_input_time = 10800 +max_execution_time = 10800 +upload_max_filesize = 1024M +post_max_size = 1024M +``` + +#### Database Configurations + ImagingUploaderAutoLaunch - This setting determines whether the insertion pipeline that archives the images and is triggered automatically or manually. @@ -87,8 +84,6 @@ MRIUploadIncomingPath - This setting determines where on the filesystem the `/data/incoming/`. This directory is created during the installation of LORIS-MRI. -patientNameRegex - Unused. - ## Interactions with LORIS From ff83bc35e668bf8dde9a3d9ec18162870153e9d9 Mon Sep 17 00:00:00 2001 From: MounaSafiHarab Date: Fri, 5 Jan 2018 14:36:47 -0500 Subject: [PATCH 07/12] second half of Daves feedback --- modules/imaging_uploader/README.md | 66 ++++++++++++++++-------------- 1 file changed, 35 insertions(+), 31 deletions(-) diff --git a/modules/imaging_uploader/README.md b/modules/imaging_uploader/README.md index d8e4165f11c..2a386bdbeb4 100644 --- a/modules/imaging_uploader/README.md +++ b/modules/imaging_uploader/README.md @@ -6,17 +6,6 @@ The imaging uploader is intended to allow users to upload, browse, and track pipeline insertion progress. -## Requirements - -For a successful upload: -1. The uploaded file is expected to be of one of the following types: -`.tgz, .tar.gz or .zip`. -2. The filename should follow the: -`PSCID_CandID_VisitLabel_*OptionalSuffix*` naming convention. -3. It is expected that the candidate and visit are already created in the -database. - - ## Intended Users The primary users are MRI technicians or site coordinators to @@ -25,20 +14,14 @@ upload imaging scans for registered LORIS candidates and timepoints. ## Scope The imaging uploader has the following built-in capabilities to facilitate -timely scan insertion into the LORIS database: - -1. Upload imaging scans using the `Upload` tab that can subsequently be inserted -into the database automatically using a collection of perl scripts provided in -[LORIS-MRI](https://github.com/aces/Loris-MRI). The launching of the insertion -can also be achieved manually -2. Browse uploaded scans using the `Browse` tab, which displays information -about who uploaded the scan, when it was uploaded, and an `UploadID` and -`UploadLocation` that can be used to trigger the LORIS-MRI insertion manually -3. Track scan insertion status through the `Progress` column and a `Log Viewer` -which displays in either `Summary` or `Detailed` mode relevant feedback messages -about insertion success or failure causes -4. Uploaded scans can be overwritten if insertion is `Not Started` or `Failed` -(i.e. if the `Progress` is neither `In Progres` nor `Success`. +timely scan insertion into the LORIS database. Specifically, it allows to browse +uploaded scans using the `Browse` tab, upload imaging scans using the `Upload` +tab, and track scan insertion status through the `Progress` column and a +`Log Viewer` which displays in either `Summary` or `Detailed` mode relevant +feedback messages about insertion success or failure causes. + +Uploaded scans can be overwritten if insertion status is `Not Started` or +`Failed` (i.e. if the `Progress` is neither `In Progress` nor `Success`). NOT in scope: @@ -50,14 +33,35 @@ are properly de-identified according to the LORIS convention. This check is however done as the first step on the LORIS-MRI side; i.e. as soon as the insertion pipeline is triggered. +## Imaging Uploader Requirements + +For a successful upload: +- The uploaded file is expected to be of one of the following types: +`.tgz`, `.tar.gz` or `.zip`. +- The filename should follow the: +`PSCID_CandID_VisitLabel_OptionalSuffix` naming convention +- It is expected that the candidate and visit are already created in the +database. + + ## Permissions +#### Module Permission + The imaging uploader module uses one permission called `imaging_uploader` that is necessary to have access to the module and gives the user the ability to upload and browse all scans uploaded to the database. +#### Filesystem Permission + +The path on the filesystem where the uploaded file go +(see section [Database Configuration](#Database Configurations)) should be +readable and writable by the web server. The LORIS-MRI install process makes it +automatically group owned by the web server. + ## Configurations + The imaging uploader has the following configurations that affect its usage: #### Install Configurations @@ -87,9 +91,9 @@ MRIUploadIncomingPath - This setting determines where on the filesystem the ## Interactions with LORIS -1. Links to other LORIS modules archived DICOMs acessible from the hyperlink in -the `TarchiveInfo` column -2. MINC images are accessible through the hyperlink in the -`Number of MincInserted` column -3. If present, MRI violated scans (as defined by the study MRI protocol) can be -accessed through the hyperlink in the `Number of MincCreated` column. +- The `TarchiveInfo` column links to the DICOM Archive module for that scan +- The `Number of MincInserted` column links to the Imaging Browser module for +that candidate's session +- The `Number of MincCreated` column links to the MRI Violated scans module if +violated scans (i.e. scans that violate the MRI protocol as defined by the +study) are present. From d177333a00dbc001fb1e44e8968a4498f3d74e2f Mon Sep 17 00:00:00 2001 From: MounaSafiHarab Date: Wed, 10 Jan 2018 13:50:45 -0500 Subject: [PATCH 08/12] add reference to the server process manager --- modules/imaging_uploader/README.md | 6 +++++- 1 file changed, 5 insertions(+), 1 deletion(-) diff --git a/modules/imaging_uploader/README.md b/modules/imaging_uploader/README.md index 2a386bdbeb4..5a50b1a3497 100644 --- a/modules/imaging_uploader/README.md +++ b/modules/imaging_uploader/README.md @@ -96,4 +96,8 @@ MRIUploadIncomingPath - This setting determines where on the filesystem the that candidate's session - The `Number of MincCreated` column links to the MRI Violated scans module if violated scans (i.e. scans that violate the MRI protocol as defined by the -study) are present. +study) are present +- If `ImagingUploaderAutoLaunch` configuration is enabled, the Server Porcess +Manager under the Admin menu can be consulted for insertion progress (exit +codes, error files, etc...). + From dfd72217f842785669d2b4bf03810ff28e7bcca2 Mon Sep 17 00:00:00 2001 From: MounaSafiHarab Date: Wed, 17 Jan 2018 13:04:18 -0500 Subject: [PATCH 09/12] Xaviers anchored suggestion and typos and better text --- modules/imaging_uploader/README.md | 25 ++++++++++++------------- 1 file changed, 12 insertions(+), 13 deletions(-) diff --git a/modules/imaging_uploader/README.md b/modules/imaging_uploader/README.md index 5a50b1a3497..0a13fb221a6 100644 --- a/modules/imaging_uploader/README.md +++ b/modules/imaging_uploader/README.md @@ -8,8 +8,8 @@ pipeline insertion progress. ## Intended Users -The primary users are MRI technicians or site coordinators to -upload imaging scans for registered LORIS candidates and timepoints. +The primary users are MRI technicians or site coordinators uploading imaging +scans for registered LORIS candidates and timepoints. ## Scope @@ -40,7 +40,7 @@ For a successful upload: `.tgz`, `.tar.gz` or `.zip`. - The filename should follow the: `PSCID_CandID_VisitLabel_OptionalSuffix` naming convention -- It is expected that the candidate and visit are already created in the +- It is expected that the candidate and timepoint are already created in the database. @@ -55,9 +55,9 @@ upload and browse all scans uploaded to the database. #### Filesystem Permission The path on the filesystem where the uploaded file go -(see section [Database Configuration](#Database Configurations)) should be -readable and writable by the web server. The LORIS-MRI install process makes it -automatically group owned by the web server. +(see section [Database Configuration](#database_config_link)) should be +readable and writable by the web server. This is automatically achieved by the +LORIS-MRI install process. ## Configurations @@ -77,14 +77,13 @@ upload_max_filesize = 1024M post_max_size = 1024M ``` -#### Database Configurations +#### Database Configurations ImagingUploaderAutoLaunch - This setting determines whether the insertion - pipeline that archives the images and is triggered automatically or - manually. + pipeline that archives the scans is triggered automatically or manually. MRIUploadIncomingPath - This setting determines where on the filesystem the - uploader is to place the uploaded file. Default location is + uploader is to place the uploaded scan. Default location is `/data/incoming/`. This directory is created during the installation of LORIS-MRI. @@ -97,7 +96,7 @@ that candidate's session - The `Number of MincCreated` column links to the MRI Violated scans module if violated scans (i.e. scans that violate the MRI protocol as defined by the study) are present -- If `ImagingUploaderAutoLaunch` configuration is enabled, the Server Porcess -Manager under the Admin menu can be consulted for insertion progress (exit -codes, error files, etc...). +- If `ImagingUploaderAutoLaunch` configuration is enabled, the Server Process +Manager under the Admin menu can be consulted for scans insertion progress +(exit codes, error files, etc...). From 46c8177bf9bfe4c348f78cfdad59f8052111285e Mon Sep 17 00:00:00 2001 From: MounaSafiHarab Date: Wed, 17 Jan 2018 16:04:54 -0500 Subject: [PATCH 10/12] Nicks feedback on removal of the scan after insertion --- modules/imaging_uploader/README.md | 5 ++++- 1 file changed, 4 insertions(+), 1 deletion(-) diff --git a/modules/imaging_uploader/README.md b/modules/imaging_uploader/README.md index 0a13fb221a6..5ee17607bb7 100644 --- a/modules/imaging_uploader/README.md +++ b/modules/imaging_uploader/README.md @@ -85,7 +85,10 @@ ImagingUploaderAutoLaunch - This setting determines whether the insertion MRIUploadIncomingPath - This setting determines where on the filesystem the uploader is to place the uploaded scan. Default location is `/data/incoming/`. This directory is created during the installation of - LORIS-MRI. + LORIS-MRI. + **Note**: Uploaded scans are erased from the + `MRIUploadIncomingPath`following a successful archival and insertion + through the LORIS-MRI pipeline. ## Interactions with LORIS From 2dd5483fd64c06a3fc59ddf4c1fba9b751f4cf49 Mon Sep 17 00:00:00 2001 From: MounaSafiHarab Date: Thu, 18 Jan 2018 14:58:19 -0500 Subject: [PATCH 11/12] ww know it is imaging uploader --- modules/imaging_uploader/README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/modules/imaging_uploader/README.md b/modules/imaging_uploader/README.md index 5ee17607bb7..3acea36b70e 100644 --- a/modules/imaging_uploader/README.md +++ b/modules/imaging_uploader/README.md @@ -33,7 +33,7 @@ are properly de-identified according to the LORIS convention. This check is however done as the first step on the LORIS-MRI side; i.e. as soon as the insertion pipeline is triggered. -## Imaging Uploader Requirements +## Requirements For a successful upload: - The uploaded file is expected to be of one of the following types: From 09bd71ebe4703d5db34158d393096d7ac1d2c4bf Mon Sep 17 00:00:00 2001 From: MounaSafiHarab Date: Mon, 22 Jan 2018 14:39:17 -0500 Subject: [PATCH 12/12] explain what those php ini settings mean --- modules/imaging_uploader/README.md | 13 +++++++------ 1 file changed, 7 insertions(+), 6 deletions(-) diff --git a/modules/imaging_uploader/README.md b/modules/imaging_uploader/README.md index 3acea36b70e..dc94a41544d 100644 --- a/modules/imaging_uploader/README.md +++ b/modules/imaging_uploader/README.md @@ -67,14 +67,15 @@ The imaging uploader has the following configurations that affect its usage: #### Install Configurations To enable the Imaging Uploader to handle large files, please update the -`php.ini` apache configuration file with the following sample values: +`php.ini` apache configuration file. Recommended sample values appropriate for +compressed scans not exceeding 500M in size are: ``` -session.gc_maxlifetime = 10800 -max_input_time = 10800 -max_execution_time = 10800 -upload_max_filesize = 1024M -post_max_size = 1024M +session.gc_maxlifetime = 10800 // After this number of seconds, stored data will be seen as 'garbage' and cleaned up by the garbage collection process. +max_input_time = 10800 // Maximum amount of time each script may spend parsing request data (in seconds) +max_execution_time = 10800 // Maximum execution time of each script (in seconds) +upload_max_filesize = 1024M // Maximum allowed size for uploaded files. +post_max_size = 1024M // Maximum size of POST data that PHP will accept. ``` #### Database Configurations