TensorBox is a simple framework for training neural networks to detect objects in images. Training requires a json file (e.g. here) containing a list of images and the bounding boxes in each image. The basic model implements the simple and robust GoogLeNet-OverFeat algorithm. We additionally provide an implementation of the ReInspect algorithm, reproducing state-of-the-art detection results on the highly occluded TUD crossing and brainwash datasets.

OverFeat Installation & Training

First, install TensorFlow from source or pip (NB: source installs currently break threading on 0.11)

$ git clone https://github.com/russell91/tensorbox
$ cd tensorbox
$ ./download_data.sh
$ cd /path/to/tensorbox && pip install -r requirements.txt
$ cd /path/to/tensorbox/utils && make && cd ..
$ python train.py --hypes hypes/overfeat_rezoom.json --gpu 0 --logdir output
$ #see evaluation instructions below

Note that running on your own dataset should only require modifying the hypes/overfeat_rezoom.json file.

ReInspect Installation & Training

ReInspect, initially implemented in Caffe, is a neural network extension to Overfeat-GoogLeNet in Tensorflow. It is designed for high performance object detection in images with heavily overlapping instances. See the paper for details or the video for a demonstration.

# REQUIRES TENSORFLOW VERSION >= 0.11
$ git clone https://github.com/russell91/tensorbox
$ cd tensorbox
$ ./download_data.sh

$ # Download the cudnn version used by your tensorflow verion and 
$ # put the libcudnn*.so files on your LD_LIBRARY_PATH e.g.
$ cp /path/to/appropriate/cudnn/lib64/* /usr/local/cuda/lib64

$ cd /path/to/tensorbox && pip install -r requirements.txt
$ cd /path/to/tensorbox/utils && make && make hungarian && cd ..
$ python train.py --hypes hypes/lstm_rezoom.json --gpu 0 --logdir output
$ #see evaluation instructions below

Evaluation

There are tree options for evaluation: an ipython notebook and two different python scripts.

IPython Notebook

The ipython notebook allows you to interactively modify the inference algorithm, and can be run concurrently with training (assuming you have 2 gpus). You can evaluate on new data by modifying paths and pointing to new weights.

Evaluation python script

For those who would prefer to evaluate using a script, you can alternately use evaluate.py. The following instructions demonstrate how evaluate.py wase used after one of my experiments - you will need to change paths as appropriate:

$ # kill training script if you don't have a spare GPU
$ cd /path/to/tensorbox
$ python evaluate.py --weights output/overfeat_rezoom_2017_01_17_15.20/save.ckpt-130000 --test_boxes data/brainwash/val_boxes.json
$ # val_boxes should contain the list of images you want to output boxes on, and
$ # the annotated boxes for each image if you want to generate a precision recall curve
$ cd ./output/overfeat_rezoom_2017_01_17_15.20/images_val_boxes_130000/
$ ls # ... notice the images with predicted boxes painted on, and the results saved in results.png
$ python -m SimpleHTTPServer 8080 # set up a image server to view the images from your browser
$ ssh myserver -N -L localhost:8080:localhost:8080 # set up an ssh tunnel to your server (skip if running locally)
$ # open firefox and visit localhost:8080 to view images

Prediction python script

Script predict.py is designed for those who prefer to use routines in your own scripts or wish to run detection for a single image only. To simply detect objects on a single image from command line run the following:

$ python predict.py image.jpg hypes.json

To write your own script which loads model and runs detection for some set of images pay attention to initialize and hot_predict routines.

Note: There are three hyperparameters which should be provided. One can do it using commandline or hyperparameters file. The hyperparameters file (hypes.json in example above) should contain the 'evaluate' section as follows:

"evaluate": {
    "gpu": 0,
    "tau": 0.2,
    "min_conf": 0.4
}

To use CPU one should omit the "gpu" option.

Finetuning

If you get some decent results and want to improve your performance, there are many things you can try. For hyperparameter optimization, the Learning rate, dropout ratios, and parameter initializations are a great place to start. You may want to read this blog post for a more generic tutorial on debugging neural nets. We have recently added a resnet version as well, which should work slightly better on larger boxes (this repo has historically done poorly on these, as they weren't port of the original research goal). I would recommend using the overfeat version over the lstm as well if you have a large variation in box sizes.

Tensorboard

You can visualize the progress of your experiments during training using Tensorboard.

$ cd /path/to/tensorbox
$ tensorboard --logdir output
$ # (optional, start an ssh tunnel if not experimenting locally)
$ ssh myserver -N -L localhost:6006:localhost:6006
$ # open localhost:6006 in your browser

For example, the following is a screenshot of a Tensorboard comparing two different experiments with learning rate decays that kick in at different points. The learning rate drops in half at 60k iterations for the green experiment and 300k iterations for red experiment.

Annotation data formats

Desription is provided in corresponding folder.

Community

If you're new to object detection, and want to chat with other people that are working on similar problems, check out the community chat at https://gitter.im/Russell91/TensorBox, especially on Saturdays.