MHP-VOS

Code for CVPR 19 paper

MHP-VOS: Multiple Hypotheses Propagation for Video Object Segmentation
[Paper] [Presentation] [Video Results]

Pipeline

In this work, we first obtain bounding box proposals from Mask RCNN, and then construct the proposal propagation tree for each object with gating and scoring strategies. To avoid calculation explosion, an N-scan pruning strategy is applied to remove branches that are far from the best hypothesis. Through this recurrent process between tree building and branches pruning, we can obtain the best propagation track, and then obtain the segmentation mask for each object by mask propagation and merging.

Introduction

we have 5 parts as following:

1. python2.7 + pycuda: pylucid
2. python2.7 + caffe: flownet2
3. python3.5 + pytorch0.3.1: mask-rcnn
4. python3.5 + pytorch0.4.1: deeplabv3+
5. Python 3.6 + pytorch0.4.1: mht

1-4 in folder libs

5 in folder mht

For more details, please refer to the README.md file in each folder

Inference

To show the perfomance of our tracking tree, we give an example inference on the carousel(test-dev) video. First of all, please download the prepare file and unzip it under the current folder. GoogleDrive BaiDuYun (code:2xwv)

The structure of prepare file looks like:

|--prepare

|----DAVIS_2017 #contains carousel video
|------Annotations
|------ImageSets
|------JPEGImages

|----deeplab_model #contains the [deeplabv3+](https://github.com/jfzhang95/pytorch-deeplab-xception) models of each carousel object
|------carousel_1_99.pth
|------carousel_2_99.pth
|------carousel_3_99.pth
|------carousel_4_99.pth

|----mask_rcnn_result #contains the bbox proposals generated from the [mask-rcnn](https://github.com/multimodallearning/pytorch-mask-rcnn)
|------carousel.json

|----osvos_result #contains the segmentation results using osvos model
|------carousel

|----test_flow #contains optical flows generated with [flownet2](https://github.com/lmb-freiburg/flownet2)
|------carousel

Run the following code:

$ cd mht
$ python test_mht.py

and it will generate three folders (vis_detections, outs and final_results)

|--mht
|----vis_detections #detection bbox
|----outs #build tree for each objects
|----final results #tracking results

Or you can just run the main.py to see all the results including the final masks in $out$ file.

$ cd mht
$ python main.py

Citation

If you use this code please cite:

@inproceedings{xu2019mhp,
    title={MHP-VOS: Multiple Hypotheses Propagation for Video Object Segmentation},
    author={Xu, Shuangjie and Liu, Daizong and Bao, Linchao and Liu, Wei and Zhou, Pan},
    booktitle={Proceedings of the IEEE conference on computer vision and pattern recognition},
    year={2019}
}