Skip to content

Commit

Permalink
remove tools dependent on mx.module APIs (apache#18508)
Browse files Browse the repository at this point in the history
* remove tools depending on mx.module

* remove caffe converter and coreml tools

Co-authored-by: Lin <[email protected]>
  • Loading branch information
2 people authored and yijunc committed Jun 9, 2020
1 parent 41bf10d commit 0bcb0be
Show file tree
Hide file tree
Showing 133 changed files with 1 addition and 14,737 deletions.
61 changes: 1 addition & 60 deletions docs/static_site/src/pages/api/faq/caffe.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,65 +26,11 @@ permalink: /api/faq/caffe

Key topics covered include the following:

- [Converting Caffe trained models to MXNet](#converting-caffe-trained-models-to-mxnet)
- [Calling Caffe operators in MXNet](#calling-caffe-operators-in-mxnet)

## Converting Caffe trained models to MXNet

The converting tool is available at
[tools/caffe_converter](https://github.com/dmlc/mxnet/tree/master/tools/caffe_converter). On
the remaining of this section, we assume we are on the `tools/caffe_converter`
directory.

### How to build

If Caffe's python package is installed, namely we can run `import caffe` in
python, then we are ready to go.

For example, we can used
[AWS Deep Learning AMI](https://aws.amazon.com/marketplace/pp/B06VSPXKDX) with
both Caffe and MXNet installed.

Otherwise we can install the
[Google protobuf](https://developers.google.com/protocol-buffers/?hl=en)
compiler and its python binding. It is easier to install, but may be slower
during running.

1. Install the compiler:
- Linux: install `protobuf-compiler` e.g. `sudo apt-get install
protobuf-compiler` for Ubuntu and `sudo yum install protobuf-compiler` for
Redhat/Fedora.
- Windows: Download the win32 build of
[protobuf](https://github.com/google/protobuf/releases). Make sure to
download the version that corresponds to the version of the python binding
on the next step. Extract to any location then add that location to your
`PATH`
- Mac OS X: `brew install protobuf`

2. Install the python binding by either `conda install -c conda-forge protobuf`
or `pip install protobuf`.

3. Compile Caffe proto definition. Run `make` in Linux or Mac OS X, or
`make_win32.bat` in Windows

### How to use

There are three tools:

- `convert_symbol.py` : convert Caffe model definition in protobuf into MXNet's
Symbol in JSON format.
- `convert_model.py` : convert Caffe model parameters into MXNet's NDArray format
- `convert_mean.py` : convert Caffe input mean file into MXNet's NDArray format

In addition, there are two tools:
- `convert_caffe_modelzoo.py` : download and convert models from Caffe model
zoo.
- `test_converter.py` : test the converted models by checking the prediction
accuracy.

## Calling Caffe operators in MXNet

Besides converting Caffe models, MXNet supports calling most Caffe operators,
MXNet supports calling most Caffe operators,
including network layer, data layer, and loss function, directly. It is
particularly useful if there are customized operators implemented in Caffe, then
we do not need to re-implement them in MXNet.
Expand Down Expand Up @@ -201,8 +147,3 @@ train = mx.io.CaffeDataIter(
num_examples = 60000,
)
```

### Put it all together

The complete example is available at
[example/caffe](https://github.com/dmlc/mxnet/blob/master/example/caffe/)
14 changes: 0 additions & 14 deletions example/ssd/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,6 @@ The arXiv paper is available [here](http://arxiv.org/abs/1512.02325).

This example is intended for reproducing the nice detector while fully utilize the
remarkable traits of MXNet.
* Model [converter](#convert-caffemodel) from caffe is available now!
* The result is almost identical to the original version. However, due to different implementation details, the results might differ slightly.

Due to the permission issue, this example is maintained in this [repository](https://github.com/zhreshold/mxnet-ssd) separately. You can use the link regarding specific per example [issues](https://github.com/zhreshold/mxnet-ssd/issues).
Expand Down Expand Up @@ -261,19 +260,6 @@ Useful when loading python symbol is not available.
python deploy.py --num-class 20
```

### Convert caffe model
Converter from caffe is available at `/path/to/incubator-mxnet/example/ssd/tools/caffe_converter`

This is specifically modified to handle custom layer in caffe-ssd. Usage:
```
cd /path/to/incubator-mxnet/example/ssd/tools/caffe_converter
make
python convert_model.py deploy.prototxt name_of_pretrained_caffe_model.caffemodel ssd_converted
# you will use this model in deploy mode without loading from python symbol(layer names inconsistent)
python demo.py --prefix ssd_converted --epoch 1 --deploy
```
There is no guarantee that conversion will always work, but at least it's good for now.

### Legacy models
Since the new interface for composing network is introduced, the old models have inconsistent names for weights.
You can still load the previous model by rename the symbol to `legacy_xxx.py`
Expand Down
5 changes: 0 additions & 5 deletions example/ssd/tools/caffe_converter/.gitignore

This file was deleted.

33 changes: 0 additions & 33 deletions example/ssd/tools/caffe_converter/Makefile

This file was deleted.

37 changes: 0 additions & 37 deletions example/ssd/tools/caffe_converter/README.md

This file was deleted.

Loading

0 comments on commit 0bcb0be

Please sign in to comment.