Skip to content

2.4 ChRIS FS DS DS plugin worflow: simulate a FreeSurfer MPC heatmap workflow

Jorge edited this page May 13, 2021 · 31 revisions

ChRIS FS->DS->DS plugin worflow: simulate a FreeSurfer / MPC / heatmap workflow

Abstract

This page provides instructions for directly interacting with the CUBE backend to create the following Feed workflow:

⬤:N     N -- Indicates the instance id of the plugin
F|DS(K)  K -- Indicates plugin id

      ⬤:1        FS(12): pl-pacscopy 
      |                  -- OR --
      |           FS(16): pl-mri10yr06mo01da_normal
      ↓
      ⬤:2        DS(14): pl-freesurfer_pp
     ╱│╲
    ╱ │ ╲
   ╱  │  ╲
  ╱   │   ╲
 ╱    │    ╲
↓     ↓     ↓ 
⬤:3  ⬤:4  ⬤:5  DS(13): pl-mpcs
│     │     │   
↓     ↓     ↓
⬤:6  ⬤:7  ⬤:8  DS(15): pl-z2labelmap

The set of operations are:

  • instantiate CUBE;
    • if using pl-mri10yr06mo01da_normal then simply run the plugin;
    • if using pl-pacscopy then after instantiation:
      • clear the internal pman job-ID jid database;
      • pull a sample dataset from github;
      • push into openstorage
      • the the pl-pacscopy plugin
  • run the FS data through pl-freesurfer_pp to create a FreeSurfer output (pre calculated on the FS output)
  • run the FreeSurfer output through three parallel pl-mpcs plugins, each generating a different output
  • run each of the 3 outputs through a pl-z2labelmap to create one of three sets of heat maps.

Instantiate CUBE

Make sure you are in the base directory of the ChRIS_ultron_backEnd (CUBE) repo:

git clone https://github.com/FNNDSC/ChRIS_ultron_backEnd
cd ChRIS_ultron_backEnd

Now run either the full CUBE instantiation with tests:

*unmake* ; sudo rm -fr CHRIS_REMOTE_FS; rm -fr CHRIS_REMOTE_FS ; *make*

or skip the unit and integration tests and the intro:

*destroy* ; sudo rm -fr CHRIS_REMOTE_FS; rm -fr CHRIS_REMOTE_FS ; *make* -U -I -s

Set some convenience environment variables

export HOST_IP=$(ip route | grep -v docker | awk '{if(NF==11) print $9}' | head -n 1)
export HOST_PORT=8000

Clear the internal pman jobid database

This step is no longer necessary since the CUBE instantiation scripts also clear the pman data base. It might however be useful to refer to here, and also if a "manual" clearing of the DB is ever necessary.

pfurl --verb POST --raw --http ${HOST_IP}:5010/api/v1/cmd \
      --jsonwrapper 'payload' --msg \
 '{  "action": "DBctl",
         "meta": {
                 "do":     "clear"
         }
 }' --quiet --jsonpprintindent 4

Create the Feed

Option 1: The easy way, using pl-mri10yr06mo01da_normal

This option entails running the pl-mri10yr06mo-1da_normal to create a Feed.

First, we need to determine the ID of the plugin. Log into the CUBE API by executing a GET on the following URL:

http://${HOST_IP}:${HOST_PORT}/api/v1/plugins/

either by using a web-client or a CLI call such as

pfurl --auth chris:chris1234 --verb GET                          \
      --http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/?limit=50    \
      --content-type application/vnd.collection+json             \
      --quiet --jsonpprintindent 4

and search for the id of the pl-mri10yr06mo01da_normal. For this example, let's assume this to be 12. Now, execute the plugin with

pfurl --auth chris:chris1234 --verb POST                          \
      --http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/12/instances/ \
      --content-type application/vnd.collection+json              \
      --jsonwrapper 'template' --msg '
{"data":
    [{"name":"dir",
      "value":"/usr/src/data"}
    ]
}' \
--quiet --jsonpprintindent 4

Note, that the instance ID of this run is 1.

Option 2: The long way, using pl-pacscopy

This option entails pulling data from the FNNDSC repo, pushing into CUBE, and then running the pl-pacscopy plugin on this data to create a Feed. It is a more involved but arguably more complete example exercise.

First, pull the sample data set that will be used in this example:

git clone https://github.com/FNNDSC/anon-10-06-01.git

Set a convenience variable:

export DICOMDIR=$(pwd)/anon-10-06-01/data

Once CUBE is up and in interactive mode, open a new terminal and cd to the CUBE repo dir (you might need to reset the convenience variables):

cd ChRIS_ultron_backEnd
export HOST_IP=$(ip route | grep -v docker | awk '{if(NF==11) print $9}')
export HOST_PORT=8000
export DICOMDIR=$(pwd)/anon-10-06-01/data

and push the data into CUBE openstorage. The PUSH operation relies on the command line apps:

  • http to PUSH via the CUBE API;
  • swift to PUSH to a swift storage container directly;

Both apps are just a pip install away

pip install httpie
pip install swift

Now, register the DICOM files in swift storage.

cd utils/scripts
./swiftCtl.sh -A push -E dcm -D $DICOMDIR -P chris/uploads/DICOM/dataset1
./swiftCtl.sh -A push -E jpg -D $DICOMDIR -P chris/uploads/DICOM/dataset1
./swiftCtl.sh -A push -E html -D $DICOMDIR -P chris/uploads/DICOM/dataset1
./swiftCtl.sh -A push -E txt -D $DICOMDIR -P chris/uploads/DICOM/dataset1
./swiftCtl.sh -A push -E json -D $DICOMDIR -P chris/uploads/DICOM/dataset1

verify the data is in fact in openstorage

./swiftCtl.sh

Now, execute the plugin with

pfurl --auth chris:chris1234 --verb POST                         \
      --http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/12/instances/ \
      --content-type application/vnd.collection+json             \
      --jsonwrapper 'template' --msg '
{"data":
    [{"name":"dir",
      "value":"DICOM/dataset1"}
    ]
}' \
--quiet --jsonpprintindent 4

Again, the choice of pl-pacscopy or pl-mri10yr06mo01da_normal should have been an either/or. In such a case, the relevant FS instance in the system has ID 1.

Query and register output files

Query the status of the plugin job:

pfurl --auth chris:chris1234                               \
      --verb GET                                           \
      --http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/1/   \
      --content-type application/vnd.collection+json       \
      --quiet --jsonpprintindent 4

Get a list of files produced by the plugin and registered to CUBE

pfurl --auth chris:chris1234                               \
      --verb GET                                           \
      --http ${HOST_IP}:${HOST_PORT}/api/v1/1/files/       \
      --content-type application/vnd.collection+json       \
      --quiet --jsonpprintindent 4

Run the DICOMs pulled in the FS instance to pl-freesurfer_pp

Next in the chain is the dummy FreeSurfer plugin, pl-freesurfer_pp that copies to its output directory the results of a pre-calculated run. In this case, the pre-populated run corresponds to the DICOM data pulled by the FS plugin.

In the CLI case, we would run the pl-freesurfer_pp plugin with

mkdir in out && chmod 777 out
docker run --rm -v $(pwd)/in:/incoming -v $(pwd)/out:/outgoing      \
        fnndsc/pl-freesurfer_pp freesurfer_pp.py                    \
        -a 10-06-01                                                 \
        -c stats,sag,cor,tra,3D                                     \
        /incoming /outgoing

In the context of CUBE, we need to determine the instance ID of the pl-freesurfer_pp, which can be found by a call to

pfurl --auth chris:chris1234 --verb GET                          \
      --http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/             \
      --content-type application/vnd.collection+json             \
      --quiet --jsonpprintindent 4

Let's assume this is plugin id 10. The equivalent call of the above CLI snippet in CUBE, using the previous id of 1 for incoming, becomes:

pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/10/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
    [
     {"name":"ageSpec",
      "value":"10-06-01"},
     {"name":"copySpec",
      "value":"stats,sag,cor,tra,3D"},
     {"name":"previous_id",
      "value":"1"}
    ]
}' \
--quiet --jsonpprintindent 4

NOTE: Please be patient with this plugin. Due to the high volume of data, it can take several minutes to complete.

Query the results of the plugin call:

pfurl --auth chris:chris1234                               \
      --verb GET                                           \
      --http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/2/   \
      --content-type application/vnd.collection+json       \
      --quiet --jsonpprintindent 4

The various MPC plugins feeding off the pl-freesurfer_pp output

Since the pl-freesurfer_pp generated a lot of files, this pl-mpcs plugin executes slowly since all the upstream volume of data needs to be transferred to the pl-mpcs. Ironically, the plugin uses none of the inputs to generate its output.

At this point, we will simulate the running of three MPC plugins that use the output generated by the dummy FreeSurfer plugin. In the figure above, we note that the plugin ID of the pl-mpcs is 11, and the instance ID of the pl-freesurfer_pp run is 2. We will now simply call the pl-mpcs plugin three times, each time with the instance id of the FreeSurfer run as previous_id:

The equivalent CLI call for the plugin would be

mkdir in out && chmod 777 out
docker run --rm -v $(pwd)/in:/incoming -v $(pwd)/out:/outgoing      \
        fnndsc/pl-mpcs mpcs.py                                      \
        --random --seed 1                                           \
        --posRange 3.0 --negRange -3.0                              \
        in out

which in the context of CUBE becomes

pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/13/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
    [
     {"name":"random",
      "value":true},
     {"name":"seed",
      "value":"1"},
     {"name":"posRange",
      "value":"3.0"},
     {"name":"negRange",
      "value":"-3.0"},
     {"name":"previous_id",
      "value":"2"}
    ]
}' \
--quiet --jsonpprintindent 4

and query the status of the call with

pfurl --auth chris:chris1234                               \
      --verb GET                                           \
      --http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/3/   \
      --content-type application/vnd.collection+json       \
      --quiet --jsonpprintindent 4

This call can take some time to complete, mostly due to data transfers.

Repeat the plugin call two more times. Test the status of each with

Instance 4

pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/13/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
    [
     {"name":"random",
      "value":true},
     {"name":"seed",
      "value":"1"},
     {"name":"posRange",
      "value":"3.0"},
     {"name":"negRange",
      "value":"-3.0"},
     {"name":"previous_id",
      "value":"2"}
    ]
}' \
--quiet --jsonpprintindent 4
pfurl --auth chris:chris1234                               \
      --verb GET                                           \
      --http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/4/   \
      --content-type application/vnd.collection+json       \
      --quiet --jsonpprintindent 4

Instance 5

pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/13/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
    [
     {"name":"random",
      "value":true},
     {"name":"seed",
      "value":"1"},
     {"name":"posRange",
      "value":"3.0"},
     {"name":"negRange",
      "value":"-3.0"},
     {"name":"previous_id",
      "value":"2"}
    ]
}' \
--quiet --jsonpprintindent 4
pfurl --auth chris:chris1234                               \
      --verb GET                                           \
      --http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/5/   \
      --content-type application/vnd.collection+json       \
      --quiet --jsonpprintindent 4

Run the pl-z2labelmap on each parallel branch

Now, finally, we will run the pl-z2labelmap on each branch of the pl-mpc output. The equivalent CLI call is

docker run --rm -v $(pwd)/in:/incoming -v $(pwd)/out:/outgoing  \
        fnndsc/pl-z2labelmap z2labelmap.py                      \
        --scaleRange 2.0 --lowerFilter 0.8                      \
        --negColor B --posColor R                               \
        /incoming /outgoing

Which, in each branch becomes:

Branch ID:6 pl-z2labelmap: connect to ID:3 pl-mpcs

pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/15/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
    [
     {"name":"imageSet",
      "value":"../data/set1"},
     {"name":"negColor",
      "value":"B"},
     {"name":"posColor",
      "value":"R"},
     {"name":"previous_id",
      "value":"3"}
    ]
}' \
--quiet --jsonpprintindent 4
pfurl --auth chris:chris1234                               \
      --verb GET                                           \
      --http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/6/   \
      --content-type application/vnd.collection+json       \
      --quiet --jsonpprintindent 4

Branch ID:7 pl-z2labelmap: connect to ID:4 pl-mpcs

pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/15/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
    [
     {"name":"imageSet",
      "value":"../data/set1"},
     {"name":"negColor",
      "value":"B"},
     {"name":"posColor",
      "value":"R"},
     {"name":"previous_id",
      "value":"4"}
    ]
}' \
--quiet --jsonpprintindent 4
pfurl --auth chris:chris1234                               \
      --verb GET                                           \
      --http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/7/   \
      --content-type application/vnd.collection+json       \
      --quiet --jsonpprintindent 4

Branch ID:8 pl-z2labelmap: connect to ID:5 pl-mpcs

pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/15/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
    [
     {"name":"imageSet",
      "value":"../data/set1"},
     {"name":"negColor",
      "value":"B"},
     {"name":"posColor",
      "value":"R"},
     {"name":"previous_id",
      "value":"5"}
    ]
}' \
--quiet --jsonpprintindent 4
pfurl --auth chris:chris1234                               \
      --verb GET                                           \
      --http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/8/   \
      --content-type application/vnd.collection+json       \
      --quiet --jsonpprintindent 4

-30-

Clone this wiki locally