Skip to content

Commit

Permalink
code release
Browse files Browse the repository at this point in the history
  • Loading branch information
zhixinwang committed Aug 27, 2019
1 parent 35a0899 commit d5a56de
Show file tree
Hide file tree
Showing 63 changed files with 87,237 additions and 14 deletions.
151 changes: 137 additions & 14 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,20 +1,143 @@
## Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection
# Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection

This respository is the code for our IROS 2019 paper [[arXiv]](https://arxiv.org/abs/1903.01864).
## Code
The code will be released within three weeks.
This repository is the code for our IROS 2019 paper [[arXiv]](https://arxiv.org/abs/1903.01864).

## Citation

If you find our work useful, please consider citing.
```
@inproceedings{wang2019frustum,
title={Frustum ConvNet: Sliding Frustums to
Aggregate Local Point-Wise Features for Amodal 3D
Object Detection},
author={Wang, Zhixin and Jia, Kui},
booktitle={IROS},
year={2019},
organization={IEEE}
If you find this work useful in your research, please consider citing.

```BibTeX
@inproceedings{wang2019frustum,
title={Frustum ConvNet: Sliding Frustums to Aggregate Local Point-Wise Features for Amodal 3D Object Detection},
author={Wang, Zhixin and Jia, Kui},
booktitle={IROS},
year={2019},
organization={IEEE}
}
```

## Installation

### Requirements

* PyTorch 1.0+
* Python 3.6+

We test our code under Ubuntu-16.04 with CUDA-9.0, CUDNN-7.0, Python-3.7.2, PyTorch-1.0.

### Clone the repository and install dependencies

```shell
git clone https://github.com/zhixinwang/frustum-convnet.git
```

You may need to install extra packages, like pybind11, opencv, yaml, tensorflow(optional).

If you want to use tensorboard to visualize the training status, you should install tensorflow (CPU version is enough).
Otherwise, you should set the config 'USE_TFBOARD: False' in cfgs/\*.yaml.

### Compile extension

```shell
cd ops
bash clean.sh
bash make.sh
```

## Download data

Download the KITTI 3D object detection dataset from [here](http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d) and organize them as follows.

```text
data/kitti
├── testing
│   ├── calib
│   ├── image_2
│   └── velodyne
└── training
├── calib
├── image_2
├── label_2
└── velodyne
```

## Training and evaluation

### First stage

Run following command to prepare pickle files for car training. We use the 2D detection results from F-PointNets.
The pickle files will be saved in `kitti/data/pickle_data`.

```shell
python kitti/prepare_data.py --car_only --gen_train --gen_val --gen_val_rgb_detection
```

Run following commands to train and evaluate the final model. You can use `export CUDA_VISIBLE_DEVICES=\?` to specify which GPU to use.
And you can modify the setting after `OUTPUT_DIR` to set a directory to save the log, model files and evaluation results. All the config settings are under the configs/config.py.

```shell
python train/train_net_det.py --cfg cfgs/det_sample.yaml OUTPUT_DIR output/car_train
python train/test_net_det.py --cfg cfgs/det_sample.yaml OUTPUT_DIR output/car_train TEST.WEIGHTS output/car_train/model_0050.pth
```

We also provide the shell script, so you can also run `bash scripts/car_train.sh` instead.

### Refinement stage

Run following command to prepare pickle files for car training. We use the first stage predicted results. If you don't use the default directory in first stage, you should change the corresponding directory in [here](kitti/prepare_data_refine.py#L911) and [here](kitti/prepare_data_refine.py#L895) before running following commands. The pickle files will be saved in `kitti/data/pickle_data_refine`.

```shell
python kitti/prepare_data_refine.py --car_only --gen_train --gen_val_det --gen_val_rgb_detection
```

Run following commands to train and evaluate the final model.

```shell
python train/train_net_det.py --cfg cfgs/refine_car.yaml OUTPUT_DIR output/car_train_refine
python train/test_net_det.py --cfg cfgs/refine_car.yaml OUTPUT_DIR output/car_train_refine TEST.WEIGHTS output/car_train_refine/model_0050.pth
```

We also provide the shell script, so you can also run `bash scripts/car_train_refine.sh` instead.

### All commands in one script file

You can simply run `bash scripts/car_all.sh` to execute all the above commands.

## Pretrained models
We provide the pretrained models for car category, you can download from [here](https://drive.google.com/open?id=1z7bBVOjtJx6qW0oKP1EcQxECqq0HP3_9).
After extracting the files under root directory, you can run `bash scripts/eval_pretrained_models.sh` to evaluate the pretrained models.
The performance on validation set is as follows:

```text
# first stage
Car [email protected], 0.70, 0.70:
bbox AP:98.33, 90.40, 88.24
bev AP:90.32, 88.02, 79.41
3d AP:87.76, 77.41, 68.79
# refinement stage
Car [email protected], 0.70, 0.70:
bbox AP:98.43, 90.39, 88.15
bev AP:90.42, 88.99, 86.88
3d AP:89.31, 79.08, 77.17
```

## Note

Since we update our code from PyTorch-0.3.1 to PyTorch-1.0 and our code uses may random sample operations, the results may be not exactly the same as these reported in our paper.
But the difference should be +-0.5\%, if you can not get the similar results, please contact me. I am still working to make results stable.

Our code is supported multiple GPUs for training, but now the training is very fast for small dataset, like KITTI, SUN-RGBD. All the steps will finish in one day on single GPU.

## TODO
- [ ] provide a demo script to visualize the detection resutls
- [ ] add the support of SUN-RGBD dataset

## Acknowledgements

Part of the code was adapted from [F-PointNets](https://github.com/charlesq34/frustum-pointnets).

## License

Our code is released under [MIT license](LICENSE).
38 changes: 38 additions & 0 deletions cfgs/det_sample.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,38 @@
TRAIN:
BATCH_SIZE: 32
START_EPOCH: 0
MAX_EPOCH: 50
OPTIMIZER: adam
BASE_LR: 0.001
MIN_LR: 1e-5
GAMMA: 0.1
LR_STEPS: [20]
MOMENTUM: 0.9
WEIGHT_DECAY: 0.0001

MODEL:
FILE: models/det_base.py
NUM_CLASSES: 2

TEST:
BATCH_SIZE: 32
METHOD: nms

DATA:
FILE: datasets/provider_sample.py
DATA_ROOT: kitti/data/pickle_data
CAR_ONLY: True
RTC: True
WITH_EXTRA_FEAT: False
NUM_SAMPLES: 1024
STRIDE: (0.25, 0.5, 1.0, 2.0)
HEIGHT_HALF: (0.25, 0.5, 1.0, 2.0)

RESUME: False
NUM_GPUS: 1
OUTPUT_DIR: 'output/car_train'
SAVE_SUB_DIR: 'val_nms'
USE_TFBOARD: True
NUM_WORKERS: 4
FROM_RGB_DET: True
disp: 100
40 changes: 40 additions & 0 deletions cfgs/refine_car.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,40 @@
TRAIN:
BATCH_SIZE: 32
START_EPOCH: 0
MAX_EPOCH: 50
OPTIMIZER: adam
BASE_LR: 0.001
MIN_LR: 1e-5
GAMMA: 0.1
LR_STEPS: [20]
MOMENTUM: 0.9
WEIGHT_DECAY: 0.0001

MODEL:
FILE: models/det_base.py
NUM_CLASSES: 2

TEST:
BATCH_SIZE: 32
METHOD: nms

DATA:
FILE: datasets/provider_sample_refine.py
DATA_ROOT: kitti/data/pickle_data_refine
CAR_ONLY: True
RTC: True
WITH_EXTRA_FEAT: False
NUM_SAMPLES: 512
STRIDE: (0.1, 0.2, 0.4, 0.8)
HEIGHT_HALF: (0.1, 0.2, 0.4, 0.8)
EXTEND_FROM_DET: False


RESUME: False
NUM_GPUS: 1
OUTPUT_DIR: 'output/car_train_refine'
SAVE_SUB_DIR: 'val_nms'
USE_TFBOARD: True
NUM_WORKERS: 4
FROM_RGB_DET: True
disp: 100
Empty file added configs/__init__.py
Empty file.
65 changes: 65 additions & 0 deletions configs/collections.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,65 @@
# Copyright (c) 2017-present, Facebook, Inc.
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
##############################################################################

"""A simple attribute dictionary used for representing configuration options."""

from __future__ import absolute_import
from __future__ import division
from __future__ import print_function
from __future__ import unicode_literals


class AttrDict(dict):
IMMUTABLE = '__immutable__'

def __init__(self, *args, **kwargs):
super(AttrDict, self).__init__(*args, **kwargs)
self.__dict__[AttrDict.IMMUTABLE] = False

def __getattr__(self, name):
if name in self.__dict__:
return self.__dict__[name]
elif name in self:
return self[name]
else:
raise AttributeError(name)

def __setattr__(self, name, value):
if not self.__dict__[AttrDict.IMMUTABLE]:
if name in self.__dict__:
self.__dict__[name] = value
else:
self[name] = value
else:
raise AttributeError(
'Attempted to set "{}" to "{}", but AttrDict is immutable'.
format(name, value)
)

def immutable(self, is_immutable):
"""Set immutability to is_immutable and recursively apply the setting
to all nested AttrDicts.
"""
self.__dict__[AttrDict.IMMUTABLE] = is_immutable
# Recursively set immutable state
for v in self.__dict__.values():
if isinstance(v, AttrDict):
v.immutable(is_immutable)
for v in self.values():
if isinstance(v, AttrDict):
v.immutable(is_immutable)

def is_immutable(self):
return self.__dict__[AttrDict.IMMUTABLE]
Loading

0 comments on commit d5a56de

Please sign in to comment.