How to retrain a pretrained network

How to retrain a pretrained network on Cloud

The following instructions use the mito_testsample dataset stored with this repository and a 30,000 iteration mitochondria trained model pre-loaded and stored in /home/ubuntu/sbem/mitochrondria/xy5.9nm40nmz/30000iterations_train_out on Cdeep3M cloud instance as an example on how to retrain a trained network.

Step 1) Connect via ssh to instance created by these instructions

ssh ubuntu@PublicDNS_VALUEFROM_WEBPAGE_ABOVE

Quick and Dirty run

Here are the commands to run once logged into the instance. Below in Steps 2+ are more detailed instructions on what is happening.

cd ~
PreprocessTrainingData.m ~/cdeep3m/mito_testsample/training/images/ ~/cdeep3m/mito_testsample/training/labels/ ~/mito_testaugtrain
runtraining.sh --additerations 200 --retrain ~/sbem/mitochrondria/xy5.9nm40nmz/30000iterations_train_out ~/mito_testaugtrain ~/mito_retrainout

Step 2) Run PreprocessTrainingData.m

More information about PreprocessTrainingData.m can be found here

cd ~
PreprocessTrainingData.m ~/cdeep3m/mito_testsample/training/images/ ~/cdeep3m/mito_testsample/training/labels/ ~/mito_testaugtrain

Output:

octave: X11 DISPLAY environment variable not set
octave: disabling GUI features
Starting Training data Preprocessing
Training Image Path:
/home/ubuntu/cdeep3m/mito_testsample/training/images/
Training Label Path:
/home/ubuntu/cdeep3m/mito_testsample/training/labels/
Output Path:
/home/ubuntu/mito_testaugtrain
Loading:
/home/ubuntu/cdeep3m/mito_testsample/training/images/
Image importer loading ...
/home/ubuntu/cdeep3m/mito_testsample/training/images/
Reading file: /home/ubuntu/cdeep3m/mito_testsample/training/images/images.010.png
.
.
/home/ubuntu/cdeep3m/mito_testsample/training/labels/
Image importer loading ...
/home/ubuntu/cdeep3m/mito_testsample/training/labels/
Reading file: /home/ubuntu/cdeep3m/mito_testsample/training/labels/mitos_3D.010.png
.
.
Create variation 8 and 16
Saving: /home/ubuntu/mito_testaugtrain/training_full_stacks_v8.h5
Elapsed time is 15.1134 seconds.
-> Training data augmentation completed
Training data stored in /home/ubuntu/mito_testaugtrain
For training your model please run CreateTrainJob.m /home/ubuntu/mito_testaugtrain <desired output directory>

Step 3) Run runtraining.sh

In this step since we already have a trained model in 30000_iterations_train_out directory so we are going to pass this model directory to runtraining.sh via the --retrain flag. We also set --additerations, in this case, to a value 200 iterations higher then what the model in 30000_iterations_train_out had been trained to.

runtraining.sh --additerations 200 --retrain ~/sbem/mitochrondria/xy5.9nm40nmz/30000iterations_train_out ~/mito_testaugtrain ~/mito_retrainout

Output:

octave: X11 DISPLAY environment variable not set
octave: disabling GUI features
Verifying input training data is valid ... success
Copying over model files and creating run scripts ... success

A new directory has been created: /home/ubuntu/mito_retrainout
In this directory are 3 directories 1fm,3fm,5fm which
correspond to 3 caffe models that need to be trained
Latest iteration found in 1fm from /home/ubuntu/sbem/mitochrondria/xy5.9nm40nmz/30000iterations_train_out is 30000
Adding 200 iterations so will now run to 30200 iterations
Copying over trained models
Copy of /home/ubuntu/sbem/mitochrondria/xy5.9nm40nmz/30000iterations_train_out/1fm/trainedmodel to /home/ubuntu/mito_retrainout/1fm/trainedmodel success
Copy of /home/ubuntu/sbem/mitochrondria/xy5.9nm40nmz/30000iterations_train_out/3fm/trainedmodel to /home/ubuntu/mito_retrainout/3fm/trainedmodel success
Copy of /home/ubuntu/sbem/mitochrondria/xy5.9nm40nmz/30000iterations_train_out/5fm/trainedmodel to /home/ubuntu/mito_retrainout/5fm/trainedmodel success

Detected 4 GPU(s). Will run in parallel.
Resuming run from snapshot file: /home/ubuntu/mito_retrainout/1fm/trainedmodel/1fm_classifer_iter_30000.solverstate
Resuming run from snapshot file: /home/ubuntu/mito_retrainout/3fm/trainedmodel/3fm_classifer_iter_30000.solverstate
Resuming run from snapshot file: /home/ubuntu/mito_retrainout/5fm/trainedmodel/5fm_classifer_iter_30000.solverstate

Training has completed. Have a nice day!


Training has completed. Results are stored in /home/ubuntu/mito_retrainout
Have a nice day!