-
Notifications
You must be signed in to change notification settings - Fork 100
2.4 ChRIS FS DS DS plugin worflow: simulate a FreeSurfer MPC heatmap workflow
This page provides instructions for directly interacting with the CUBE backend to create the following Feed workflow:
⬤:N N -- Indicates the instance id of the plugin
F|DS(K) K -- Indicates plugin id
⬤:1 FS(12): pl-pacscopy
| -- OR --
| FS(16): pl-mri10yr06mo01da_normal
↓
⬤:2 DS(14): pl-freesurfer_pp
╱│╲
╱ │ ╲
╱ │ ╲
╱ │ ╲
╱ │ ╲
↓ ↓ ↓
⬤:3 ⬤:4 ⬤:5 DS(13): pl-mpcs
│ │ │
↓ ↓ ↓
⬤:6 ⬤:7 ⬤:8 DS(15): pl-z2labelmap
The set of operations are:
- instantiate CUBE;
- if using
pl-mri10yr06mo01da_normal
then simply run the plugin; - if using
pl-pacscopy
then after instantiation:- clear the internal
pman
job-IDjid
database; - pull a sample dataset from github;
- push into openstorage
- the the
pl-pacscopy
plugin
- clear the internal
- if using
- run the FS data through
pl-freesurfer_pp
to create a FreeSurfer output (pre calculated on the FS output) - run the FreeSurfer output through three parallel
pl-mpcs
plugins, each generating a different output - run each of the 3 outputs through a
pl-z2labelmap
to create one of three sets of heat maps.
Make sure you are in the base directory of the ChRIS_ultron_backEnd (CUBE) repo:
git clone https://github.com/FNNDSC/ChRIS_ultron_backEnd
cd ChRIS_ultron_backEnd
Now run either the full CUBE instantiation with tests:
*unmake* ; sudo rm -fr CHRIS_REMOTE_FS; rm -fr CHRIS_REMOTE_FS ; *make*
or skip the unit and integration tests and the intro:
*destroy* ; sudo rm -fr CHRIS_REMOTE_FS; rm -fr CHRIS_REMOTE_FS ; *make* -U -I -s
export HOST_IP=$(ip route | grep -v docker | awk '{if(NF==11) print $9}' | head -n 1)
export HOST_PORT=8000
This step is no longer necessary since the CUBE instantiation scripts also clear the pman
data base. It might however be useful to refer to here, and also if a "manual" clearing of the DB is ever necessary.
pfurl --verb POST --raw --http ${HOST_IP}:5010/api/v1/cmd \
--jsonwrapper 'payload' --msg \
'{ "action": "DBctl",
"meta": {
"do": "clear"
}
}' --quiet --jsonpprintindent 4
This option entails running the pl-mri10yr06mo-1da_normal
to create a Feed.
First, we need to determine the ID of the plugin. Log into the CUBE API by executing a GET on the following URL:
http://${HOST_IP}:${HOST_PORT}/api/v1/plugins/
either by using a web-client or a CLI call such as
pfurl --auth chris:chris1234 --verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/?limit=50 \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4
and search for the id
of the pl-mri10yr06mo01da_normal
. For this example, let's assume this to be 12
. Now, execute the plugin with
pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/12/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
[{"name":"dir",
"value":"/usr/src/data"}
]
}' \
--quiet --jsonpprintindent 4
Note, that the instance ID of this run is 1
.
This option entails pulling data from the FNNDSC repo, pushing into CUBE, and then running the pl-pacscopy
plugin on this data to create a Feed. It is a more involved but arguably more complete example exercise.
First, pull the sample data set that will be used in this example:
git clone https://github.com/FNNDSC/anon-10-06-01.git
Set a convenience variable:
export DICOMDIR=$(pwd)/anon-10-06-01/data
Once CUBE is up and in interactive mode, open a new terminal and cd
to the CUBE repo dir (you might need to reset the convenience variables):
cd ChRIS_ultron_backEnd
export HOST_IP=$(ip route | grep -v docker | awk '{if(NF==11) print $9}')
export HOST_PORT=8000
export DICOMDIR=$(pwd)/anon-10-06-01/data
and push the data into CUBE openstorage. The PUSH operation relies on the command line apps:
-
http
to PUSH via the CUBE API; -
swift
to PUSH to a swift storage container directly;
Both apps are just a pip install
away
pip install httpie
pip install swift
Now, register the DICOM files in swift storage.
cd utils/scripts
./swiftCtl.sh -A push -E dcm -D $DICOMDIR -P chris/uploads/DICOM/dataset1
./swiftCtl.sh -A push -E jpg -D $DICOMDIR -P chris/uploads/DICOM/dataset1
./swiftCtl.sh -A push -E html -D $DICOMDIR -P chris/uploads/DICOM/dataset1
./swiftCtl.sh -A push -E txt -D $DICOMDIR -P chris/uploads/DICOM/dataset1
./swiftCtl.sh -A push -E json -D $DICOMDIR -P chris/uploads/DICOM/dataset1
verify the data is in fact in openstorage
./swiftCtl.sh
Now, execute the plugin with
pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/12/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
[{"name":"dir",
"value":"DICOM/dataset1"}
]
}' \
--quiet --jsonpprintindent 4
Again, the choice of pl-pacscopy
or pl-mri10yr06mo01da_normal
should have been an either/or. In such a case, the relevant FS instance in the system has ID 1
.
Query the status of the plugin job:
pfurl --auth chris:chris1234 \
--verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/1/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4
pfurl --auth chris:chris1234 \
--verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/1/files/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4
Next in the chain is the dummy FreeSurfer plugin, pl-freesurfer_pp
that copies to its output directory the results of a pre-calculated run. In this case, the pre-populated run corresponds to the DICOM data pulled by the FS plugin.
In the CLI case, we would run the pl-freesurfer_pp
plugin with
mkdir in out && chmod 777 out
docker run --rm -v $(pwd)/in:/incoming -v $(pwd)/out:/outgoing \
fnndsc/pl-freesurfer_pp freesurfer_pp.py \
-a 10-06-01 \
-c stats,sag,cor,tra,3D \
/incoming /outgoing
In the context of CUBE, we need to determine the instance ID of the pl-freesurfer_pp
, which can be found by a call to
pfurl --auth chris:chris1234 --verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4
Let's assume this is plugin id 10
. The equivalent call of the above CLI snippet in CUBE, using the previous id of 1
for incoming
, becomes:
pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/10/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
[
{"name":"ageSpec",
"value":"10-06-01"},
{"name":"copySpec",
"value":"stats,sag,cor,tra,3D"},
{"name":"previous_id",
"value":"1"}
]
}' \
--quiet --jsonpprintindent 4
NOTE: Please be patient with this plugin. Due to the high volume of data, it can take several minutes to complete.
Query the results of the plugin call:
pfurl --auth chris:chris1234 \
--verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/2/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4
Since the pl-freesurfer_pp
generated a lot of files, this pl-mpcs
plugin executes slowly since all the upstream volume of data needs to be transferred to the pl-mpcs
. Ironically, the plugin uses none of the inputs to generate its output.
At this point, we will simulate the running of three MPC plugins that use the output generated by the dummy FreeSurfer plugin. In the figure above, we note that the plugin ID of the pl-mpcs
is 11, and the instance ID of the pl-freesurfer_pp
run is 2
. We will now simply call the pl-mpcs
plugin three times, each time with the instance id of the FreeSurfer run as previous_id
:
The equivalent CLI call for the plugin would be
mkdir in out && chmod 777 out
docker run --rm -v $(pwd)/in:/incoming -v $(pwd)/out:/outgoing \
fnndsc/pl-mpcs mpcs.py \
--random --seed 1 \
--posRange 3.0 --negRange -3.0 \
in out
which in the context of CUBE becomes
pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/13/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
[
{"name":"random",
"value":true},
{"name":"seed",
"value":"1"},
{"name":"posRange",
"value":"3.0"},
{"name":"negRange",
"value":"-3.0"},
{"name":"previous_id",
"value":"2"}
]
}' \
--quiet --jsonpprintindent 4
and query the status of the call with
pfurl --auth chris:chris1234 \
--verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/3/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4
This call can take some time to complete, mostly due to data transfers.
Repeat the plugin call two more times. Test the status of each with
pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/13/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
[
{"name":"random",
"value":true},
{"name":"seed",
"value":"1"},
{"name":"posRange",
"value":"3.0"},
{"name":"negRange",
"value":"-3.0"},
{"name":"previous_id",
"value":"2"}
]
}' \
--quiet --jsonpprintindent 4
pfurl --auth chris:chris1234 \
--verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/4/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4
pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/13/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
[
{"name":"random",
"value":true},
{"name":"seed",
"value":"1"},
{"name":"posRange",
"value":"3.0"},
{"name":"negRange",
"value":"-3.0"},
{"name":"previous_id",
"value":"2"}
]
}' \
--quiet --jsonpprintindent 4
pfurl --auth chris:chris1234 \
--verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/5/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4
Now, finally, we will run the pl-z2labelmap
on each branch of the pl-mpc
output. The equivalent CLI call is
docker run --rm -v $(pwd)/in:/incoming -v $(pwd)/out:/outgoing \
fnndsc/pl-z2labelmap z2labelmap.py \
--scaleRange 2.0 --lowerFilter 0.8 \
--negColor B --posColor R \
/incoming /outgoing
Which, in each branch becomes:
pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/15/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
[
{"name":"imageSet",
"value":"../data/set1"},
{"name":"negColor",
"value":"B"},
{"name":"posColor",
"value":"R"},
{"name":"previous_id",
"value":"3"}
]
}' \
--quiet --jsonpprintindent 4
pfurl --auth chris:chris1234 \
--verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/6/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4
pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/15/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
[
{"name":"imageSet",
"value":"../data/set1"},
{"name":"negColor",
"value":"B"},
{"name":"posColor",
"value":"R"},
{"name":"previous_id",
"value":"4"}
]
}' \
--quiet --jsonpprintindent 4
pfurl --auth chris:chris1234 \
--verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/7/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4
pfurl --auth chris:chris1234 --verb POST \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/15/instances/ \
--content-type application/vnd.collection+json \
--jsonwrapper 'template' --msg '
{"data":
[
{"name":"imageSet",
"value":"../data/set1"},
{"name":"negColor",
"value":"B"},
{"name":"posColor",
"value":"R"},
{"name":"previous_id",
"value":"5"}
]
}' \
--quiet --jsonpprintindent 4
pfurl --auth chris:chris1234 \
--verb GET \
--http ${HOST_IP}:${HOST_PORT}/api/v1/plugins/instances/8/ \
--content-type application/vnd.collection+json \
--quiet --jsonpprintindent 4
-30-