Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

docs: add model support #784

Merged
merged 13 commits into from
Aug 2, 2022
Merged

docs: add model support #784

Changes from 8 commits
Commits
Show all changes
13 commits
Select commit Hold shift + click to select a range
4cd5b40
docs: add model support
shan-mx Jul 26, 2022
0e45928
docs: fix name
shan-mx Jul 26, 2022
e025af3
docs: clean models names
numb3r3 Jul 29, 2022
ae76902
fix: polish model and weight names
numb3r3 Aug 1, 2022
9a483e1
docs: revision
numb3r3 Aug 1, 2022
8675ec1
docs: revision
numb3r3 Aug 1, 2022
92f63be
docs: update benchmark table
ZiniuYu Aug 2, 2022
4ed805c
docs: update benchmark table and narratives
ZiniuYu Aug 2, 2022
87f3ec4
docs: refactor ref
ZiniuYu Aug 2, 2022
b6102a3
docs: refactor ref
ZiniuYu Aug 2, 2022
9bf86f8
docs: typo
ZiniuYu Aug 2, 2022
e2c4d7b
docs: typo
ZiniuYu Aug 2, 2022
58d5eb4
docs: fix link
ZiniuYu Aug 2, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
51 changes: 33 additions & 18 deletions docs/user-guides/server.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -61,23 +61,38 @@ The procedure and UI of ONNX and TensorRT runtime would look the same as Pytorch

ZiniuYu marked this conversation as resolved.
Show resolved Hide resolved
## Model support

Open AI has released 9 models so far. `ViT-B/32` is used as default model in all runtimes. Due to the limitation of some runtime, not every runtime supports all nine models. Please also note that different model give different size of output dimensions. This will affect your downstream applications. For example, switching the model from one to another make your embedding incomparable, which breaks the downstream applications. Below is a list of supported models of each runtime and its corresponding size. We include the disk usage (in delta) and the peak RAM and VRAM usage (in delta) when running on a single Nvidia TITAN RTX GPU (24GB VRAM) for a series of text and image encoding tasks with `batch_size=8` using PyTorch runtime.

| Model | PyTorch | ONNX | TensorRT | Output Dimension | Disk Usage (MB) | Peak RAM Usage (GB) | Peak VRAM Usage (GB) |
|----------------|---------|------|----------|------------------|-----------------|---------------------|----------------------|
| RN50 | ✅ | ✅ | ✅ | 1024 | 256 | 2.99 | 1.36 |
| RN101 | ✅ | ✅ | ✅ | 512 | 292 | 3.51 | 1.40 |
| RN50x4 | ✅ | ✅ | ✅ | 640 | 422 | 3.23 | 1.63 |
| RN50x16 | ✅ | ✅ | ❌ | 768 | 661 | 3.63 | 2.02 |
| RN50x64 | ✅ | ✅ | ❌ | 1024 | 1382 | 4.08 | 2.98 |
| ViT-B/32 | ✅ | ✅ | ✅ | 512 | 351 | 3.20 | 1.40 |
| ViT-B/16 | ✅ | ✅ | ✅ | 512 | 354 | 3.20 | 1.44 |
| ViT-L/14 | ✅ | ✅ | ❌ | 768 | 933 | 3.66 | 2.04 |
| ViT-L/14@336px | ✅ | ✅ | ❌ | 768 | 934 | 3.74 | 2.23 |

### Use custom model

You can also use your own model in ONNX runtime by specifying the model name and the path to model directory in YAML file.
The various `CLIP` models implemented in the [OpenAI](https://github.com/openai/CLIP), [OpenCLIP](https://github.com/mlfoundations/open_clip), and [MultilingualCLIP](https://github.com/FreddeFrallan/Multilingual-CLIP) are supported.
Due to the limitation of some runtimes, not every runtime supports all models.
Please also note that **different models give different sizes of output dimensions**. This will affect your downstream applications. For example, switching the model from one to another make your embedding incomparable, which breaks the downstream applications. Below is a list of supported models of each runtime and its corresponding size. We include the disk usage (in delta) and the peak RAM and VRAM usage (in delta) when running on a single Nvidia TITAN RTX GPU (24GB VRAM) for a series of text and image encoding tasks with `batch_size=8` using PyTorch runtime.

| Model | PyTorch | ONNX | TensorRT | Output Dimension | Disk Usage (MB) | Peak RAM Usage (GB) | Peak VRAM Usage (GB) |
|---------------------------------------|---------|------|----------|------------------|-----------------|---------------------|----------------------|
| RN50 | ✅ | ✅ | ✅ | 1024 | 244 | 2.99 | 1.36 |
| RN101 | ✅ | ✅ | ✅ | 512 | 278 | 3.05 | 1.40 |
| RN50x4 | ✅ | ✅ | ✅ | 640 | 402 | 3.23 | 1.63 |
| RN50x16 | ✅ | ✅ | ❌ | 768 | 631 | 3.63 | 2.02 |
| RN50x64 | ✅ | ✅ | ❌ | 1024 | 1291 | 4.08 | 2.98 |
| ViT-B-32 | ✅ | ✅ | ✅ | 512 | 338 | 3.20 | 1.40 |
| ViT-B-16 | ✅ | ✅ | ✅ | 512 | 335 | 3.20 | 1.44 |
| ViT-B-16-plus-240 | ✅ | ✅ | 🚧 | 640 | 795 | 3.03 | 1.59 |
| ViT-L-14 | ✅ | ✅ | ❌ | 768 | 890 | 3.66 | 2.04 |
| ViT-L-14@336px | ✅ | ✅ | ❌ | 768 | 891 | 3.74 | 2.23 |
| M-CLIP/XLM-Roberta-Large-Vit-B-32 | ✅ | 🚧 | 🚧 | 512 | 4284 | 5.37 | 1.68 |
| M-CLIP/XLM-Roberta-Large-Vit-L-14 | ✅ | 🚧 | ❌ | 768 | 4293 | 4.30 | 4.97 |
| M-CLIP/XLM-Roberta-Large-Vit-B-16Plus | ✅ | 🚧 | 🚧 | 640 | 4293 | 4.30 | 4.13 |
| M-CLIP/LABSE-Vit-L-14 | ✅ | 🚧 | ❌ | 768 | 3609 | 4.30 | 4.70 |

✅ = First class support — 🚧 = Unsupported, working in progress

`ViT-B-32::openai` is used as the default model in all runtimes. To use specific pretrained models provided by `open_clip`, please use `::` to separate model name and pretrained weight name, e.g. `ViT-B-32::laion2b_e16`.
Full list of open_clip models and weights can be found [here](https://github.com/mlfoundations/open_clip#pretrained-model-interface).

```{note}
For model definition with `-quickgelu` postfix, please use non `-quickgelu` model name.
```

### Use custom model for onnx
You can also use your own model in ONNX runtime by specifying the model name and the path to ONNX model directory in YAML file.
The model directory should have the same structure as below:

```text
Expand Down Expand Up @@ -249,7 +264,7 @@ For all backends, you can set the following parameters via `with`:

| Parameter | Description |
|-------------------------|--------------------------------------------------------------------------------------------------------------------------------|
| `name` | Model weights, default is `ViT-B/32`. Support all OpenAI released pretrained models. |
| `name` | Model weights, default is `ViT-B-32::openai`. Full list of models and weights can be found at [here](#model-support) |
ZiniuYu marked this conversation as resolved.
Show resolved Hide resolved
| `num_worker_preprocess` | The number of CPU workers for image & text prerpocessing, default 4. |
| `minibatch_size` | The size of a minibatch for CPU preprocessing and GPU encoding, default 64. Reduce the size of it if you encounter OOM on GPU. |

Expand Down