Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/en/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ lmdeploy serve api_server internlm/internlm2_5-7b-chat --cache-max-entry-count 0
### Api Server Fetch Timeout

The image URL fetch timeout for the API server can be configured via the environment variable `LMDEPLOY_FETCH_TIMEOUT`.
By default, requests may take up to 10 seconds before timing out. See [lmdeploy/vl/utils.py](https://github.com/InternLM/lmdeploy/blob/7b6876eafcb842633e0efe8baabe5906d7beeeea/lmdeploy/vl/utils.py#L31) for usage.
By default, requests may take up to 10 seconds before timing out. See [lmdeploy/multimodal/utils.py](https://github.com/InternLM/lmdeploy/blob/main/lmdeploy/multimodal/utils.py) for usage.

## Quantization

Expand Down
2 changes: 1 addition & 1 deletion docs/en/get_started/ascend/get_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -50,7 +50,7 @@ Set `device_type="ascend"` in the `PytorchEngineConfig`:

```python
from lmdeploy import pipeline, PytorchEngineConfig
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image
pipe = pipeline('OpenGVLab/InternVL2-2B',
backend_config=PytorchEngineConfig(tp=1, device_type='ascend'))
image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
Expand Down
2 changes: 1 addition & 1 deletion docs/en/get_started/camb/get_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ Set `device_type="camb"` in the `PytorchEngineConfig`:

```python
from lmdeploy import pipeline, PytorchEngineConfig
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image
pipe = pipeline('OpenGVLab/InternVL2-2B',
backend_config=PytorchEngineConfig(tp=1, device_type='camb'))
image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
Expand Down
4 changes: 2 additions & 2 deletions docs/en/get_started/get_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ For example, you can utilize the following code snippet to perform the inference

```python
from lmdeploy import pipeline
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image

pipe = pipeline('OpenGVLab/InternVL2-8B')

Expand All @@ -96,7 +96,7 @@ In VLM pipeline, the default image processing batch size is 1. This can be adjus

```python
from lmdeploy import pipeline, VisionConfig
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image

pipe = pipeline('OpenGVLab/InternVL2-8B',
vision_config=VisionConfig(
Expand Down
2 changes: 1 addition & 1 deletion docs/en/get_started/maca/get_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ Set `device_type="maca"` in the `PytorchEngineConfig`:

```python
from lmdeploy import pipeline, PytorchEngineConfig
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image
pipe = pipeline('OpenGVLab/InternVL2-2B',
backend_config=PytorchEngineConfig(tp=1, device_type='maca'))
image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
Expand Down
2 changes: 1 addition & 1 deletion docs/en/multi_modal/cogvlm.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ The following sample code shows the basic usage of VLM pipeline. For more exampl

```python
from lmdeploy import pipeline
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image


if __name__ == "__main__":
Expand Down
2 changes: 1 addition & 1 deletion docs/en/multi_modal/deepseek_vl2.md
Original file line number Diff line number Diff line change
Expand Up @@ -30,7 +30,7 @@ To construct valid DeepSeek-VL2 prompts with image inputs, users should insert `

```python
from lmdeploy import pipeline
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image


if __name__ == "__main__":
Expand Down
2 changes: 1 addition & 1 deletion docs/en/multi_modal/gemma3.md
Original file line number Diff line number Diff line change
Expand Up @@ -18,7 +18,7 @@ The following sample code shows the basic usage of VLM pipeline. For more exampl

```python
from lmdeploy import pipeline
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image


if __name__ == "__main__":
Expand Down
5 changes: 4 additions & 1 deletion docs/en/multi_modal/index.rst
Original file line number Diff line number Diff line change
@@ -1,6 +1,9 @@
Vision-Language Models
Multimodal Models
=================================

Use ``lmdeploy.multimodal`` for multimodal helper APIs such as media loading
and local-file encoding.

.. toctree::
:maxdepth: 2
:caption: Examples
Expand Down
10 changes: 5 additions & 5 deletions docs/en/multi_modal/internvl.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ The following sample code shows the basic usage of VLM pipeline. For detailed in

```python
from lmdeploy import pipeline
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image

pipe = pipeline('OpenGVLab/InternVL2-8B')

Expand All @@ -60,7 +60,7 @@ More examples are listed below:

```python
from lmdeploy import pipeline, GenerationConfig
from lmdeploy.vl.constants import IMAGE_TOKEN
from lmdeploy.multimodal.constants import IMAGE_TOKEN

pipe = pipeline('OpenGVLab/InternVL2-8B', log_level='INFO')
messages = [
Expand All @@ -86,7 +86,7 @@ out = pipe(messages, gen_config=GenerationConfig(top_k=1))

```python
from lmdeploy import pipeline, GenerationConfig
from lmdeploy.vl.constants import IMAGE_TOKEN
from lmdeploy.multimodal.constants import IMAGE_TOKEN

pipe = pipeline('OpenGVLab/InternVL2-8B', log_level='INFO')
messages = [
Expand Down Expand Up @@ -114,8 +114,8 @@ out = pipe(messages, gen_config=GenerationConfig(top_k=1))
import numpy as np
from lmdeploy import pipeline, GenerationConfig
from decord import VideoReader, cpu
from lmdeploy.vl.constants import IMAGE_TOKEN
from lmdeploy.vl import encode_image_base64
from lmdeploy.multimodal.constants import IMAGE_TOKEN
from lmdeploy.multimodal import encode_image_base64
from PIL import Image
pipe = pipeline('OpenGVLab/InternVL2-8B', log_level='INFO')

Expand Down
2 changes: 1 addition & 1 deletion docs/en/multi_modal/llava.md
Original file line number Diff line number Diff line change
Expand Up @@ -33,7 +33,7 @@ The following sample code shows the basic usage of VLM pipeline. For detailed in

```python
from lmdeploy import GenerationConfig, TurbomindEngineConfig, pipeline
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image


pipe = pipeline("llava-hf/llava-interleave-qwen-7b-hf", backend_config=TurbomindEngineConfig(cache_max_entry_count=0.5),
Expand Down
4 changes: 2 additions & 2 deletions docs/en/multi_modal/minicpmv.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ The following sample code shows the basic usage of VLM pipeline. For detailed in

```python
from lmdeploy import pipeline
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image

pipe = pipeline('openbmb/MiniCPM-V-2_6')

Expand Down Expand Up @@ -97,7 +97,7 @@ print(out.text)

```python
from lmdeploy import pipeline, GenerationConfig
from lmdeploy.vl import encode_image_base64
from lmdeploy.multimodal import encode_image_base64
import torch
from PIL import Image
from transformers import AutoModel, AutoTokenizer
Expand Down
2 changes: 1 addition & 1 deletion docs/en/multi_modal/molmo.md
Original file line number Diff line number Diff line change
Expand Up @@ -19,7 +19,7 @@ The following sample code shows the basic usage of VLM pipeline. For detailed in

```python
from lmdeploy import pipeline
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image

pipe = pipeline('allenai/Molmo-7B-D-0924')

Expand Down
10 changes: 5 additions & 5 deletions docs/en/multi_modal/multimodal_inputs.md
Original file line number Diff line number Diff line change
Expand Up @@ -398,7 +398,7 @@ In addition to HTTP URLs, lmdeploy accepts:
- **Local file paths** via `file://` scheme: `file:///absolute/path/to/file.jpg`
- **Base64-encoded data** via data URLs: `data:<mime>;base64,<encoded_data>`

Use the helpers in `lmdeploy.vl.utils` to encode local files:
Use the helpers in `lmdeploy.multimodal.utils` to encode local files:

<details>
<summary>Local file path example</summary>
Expand Down Expand Up @@ -434,7 +434,7 @@ print(response.choices[0].message.content)

```python
from openai import OpenAI
from lmdeploy.vl.utils import encode_image_base64
from lmdeploy.multimodal.utils import encode_image_base64

client = OpenAI(api_key='EMPTY', base_url='http://localhost:23333/v1')
model_name = client.models.list().data[0].id
Expand Down Expand Up @@ -465,7 +465,7 @@ print(response.choices[0].message.content)

```python
from openai import OpenAI
from lmdeploy.vl.utils import encode_video_base64
from lmdeploy.multimodal.utils import encode_video_base64

client = OpenAI(api_key='EMPTY', base_url='http://localhost:23333/v1')
model_name = client.models.list().data[0].id
Expand Down Expand Up @@ -497,7 +497,7 @@ print(response.choices[0].message.content)

```python
from openai import OpenAI
from lmdeploy.vl.utils import encode_audio_base64
from lmdeploy.multimodal.utils import encode_audio_base64

client = OpenAI(api_key='EMPTY', base_url='http://localhost:23333/v1')
model_name = client.models.list().data[0].id
Expand Down Expand Up @@ -528,7 +528,7 @@ print(response.choices[0].message.content)

```python
from openai import OpenAI
from lmdeploy.vl.utils import encode_time_series_base64
from lmdeploy.multimodal.utils import encode_time_series_base64

client = OpenAI(api_key='EMPTY', base_url='http://localhost:23333/v1')
model_name = client.models.list().data[0].id
Expand Down
2 changes: 1 addition & 1 deletion docs/en/multi_modal/phi3.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,7 @@ The following sample code shows the basic usage of VLM pipeline. For more exampl

```python
from lmdeploy import pipeline
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image

pipe = pipeline('microsoft/Phi-3.5-vision-instruct')

Expand Down
6 changes: 3 additions & 3 deletions docs/en/multi_modal/qwen2_5_vl.md
Original file line number Diff line number Diff line change
Expand Up @@ -25,7 +25,7 @@ The following sample code shows the basic usage of the VLM pipeline. For detaile

```python
from lmdeploy import pipeline
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image

pipe = pipeline('Qwen/Qwen2.5-VL-7B-Instruct')

Expand Down Expand Up @@ -98,8 +98,8 @@ out = pipe(messages, gen_config=GenerationConfig(top_k=1))
import numpy as np
from lmdeploy import pipeline, GenerationConfig
from decord import VideoReader, cpu
from lmdeploy.vl.constants import IMAGE_TOKEN
from lmdeploy.vl import encode_image_base64
from lmdeploy.multimodal.constants import IMAGE_TOKEN
from lmdeploy.multimodal import encode_image_base64
from PIL import Image
pipe = pipeline('Qwen/Qwen2.5-VL-7B-Instruct', log_level='INFO')

Expand Down
2 changes: 1 addition & 1 deletion docs/en/multi_modal/qwen2_vl.md
Original file line number Diff line number Diff line change
Expand Up @@ -36,7 +36,7 @@ The following sample code shows the basic usage of VLM pipeline. For detailed in

```python
from lmdeploy import pipeline
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image

pipe = pipeline('Qwen/Qwen2-VL-2B-Instruct')

Expand Down
26 changes: 13 additions & 13 deletions docs/en/multi_modal/vl_pipeline.md
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@ Using the pipeline interface to infer other VLM models is similar, with the main

```python
from lmdeploy import pipeline
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image

pipe = pipeline('OpenGVLab/InternVL2_5-8B')

Expand Down Expand Up @@ -53,7 +53,7 @@ Tensor paramllelism can be activated by setting the engine parameter `tp`

```python
from lmdeploy import pipeline, TurbomindEngineConfig
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image

pipe = pipeline('OpenGVLab/InternVL2_5-8B',
backend_config=TurbomindEngineConfig(tp=2))
Expand All @@ -69,7 +69,7 @@ When creating the pipeline, you can customize the size of the context window by

```python
from lmdeploy import pipeline, TurbomindEngineConfig
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image

pipe = pipeline('OpenGVLab/InternVL2_5-8B',
backend_config=TurbomindEngineConfig(session_len=8192))
Expand All @@ -85,7 +85,7 @@ You can change the default sampling parameters of pipeline by passing `Generatio

```python
from lmdeploy import pipeline, GenerationConfig, TurbomindEngineConfig
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image

pipe = pipeline('OpenGVLab/InternVL2_5-8B',
backend_config=TurbomindEngineConfig(tp=2, session_len=8192))
Expand All @@ -101,8 +101,8 @@ By default, LMDeploy inserts the special image token into the user prompt follow

```python
from lmdeploy import pipeline
from lmdeploy.vl import load_image
from lmdeploy.vl.constants import IMAGE_TOKEN
from lmdeploy.multimodal import load_image
from lmdeploy.multimodal.constants import IMAGE_TOKEN

pipe = pipeline('deepseek-ai/deepseek-vl-1.3b-chat')

Expand All @@ -117,7 +117,7 @@ While performing inference, LMDeploy identifies an appropriate chat template fro

```python
from lmdeploy import pipeline, ChatTemplateConfig
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image
pipe = pipeline('local_model_folder',
chat_template_config=ChatTemplateConfig(model_name='llava-v1'))
image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
Expand All @@ -133,7 +133,7 @@ The default parameters of the visual model can be modified by setting `VisionCon

```python
from lmdeploy import pipeline, VisionConfig
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image
vision_config=VisionConfig(max_batch_size=16)
pipe = pipeline('liuhaotian/llava-v1.5-7b', vision_config=vision_config)
image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
Expand All @@ -145,7 +145,7 @@ print(response)

```python
from lmdeploy import pipeline, GenerationConfig
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image
pipe = pipeline('OpenGVLab/InternVL2_5-8B')

image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
Expand All @@ -162,7 +162,7 @@ When dealing with multiple images, you can put them all in one list. Keep in min

```python
from lmdeploy import pipeline, TurbomindEngineConfig
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image

pipe = pipeline('OpenGVLab/InternVL2_5-8B',
backend_config=TurbomindEngineConfig(session_len=8192))
Expand All @@ -183,7 +183,7 @@ Conducting inference with batch prompts is quite straightforward; just place the

```python
from lmdeploy import pipeline, TurbomindEngineConfig
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image

pipe = pipeline('OpenGVLab/InternVL2_5-8B',
backend_config=TurbomindEngineConfig(session_len=8192))
Expand All @@ -203,7 +203,7 @@ There are two ways to do the multi-turn conversations with the pipeline. One is

```python
from lmdeploy import pipeline, TurbomindEngineConfig, GenerationConfig
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image

pipe = pipeline('OpenGVLab/InternVL2_5-8B',
backend_config=TurbomindEngineConfig(session_len=8192))
Expand All @@ -224,7 +224,7 @@ You can release the pipeline explicitly by calling its `close()` method, or alte
from lmdeploy import pipeline

from lmdeploy import pipeline
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image

with pipeline('OpenGVLab/InternVL2_5-8B') as pipe:
image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
Expand Down
2 changes: 1 addition & 1 deletion docs/zh_cn/faq.md
Original file line number Diff line number Diff line change
Expand Up @@ -94,7 +94,7 @@ lmdeploy serve api_server internlm/internlm2_5-7b-chat --cache-max-entry-count 0

API 服务器的图像 URL 获取超时可通过环境变量 `LMDEPLOY_FETCH_TIMEOUT` 进行配置。默认情况下,请求可能需要长达 10 秒才会超时。

请参阅 [lmdeploy/vl/utils.py](https://github.com/InternLM/lmdeploy/blob/7b6876eafcb842633e0efe8baabe5906d7beeeea/lmdeploy/vl/utils.py#L31) 了解用法。
请参阅 [lmdeploy/multimodal/utils.py](https://github.com/InternLM/lmdeploy/blob/main/lmdeploy/multimodal/utils.py) 了解用法。

## 量化

Expand Down
2 changes: 1 addition & 1 deletion docs/zh_cn/get_started/ascend/get_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -43,7 +43,7 @@ print(response)

```python
from lmdeploy import pipeline, PytorchEngineConfig
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image
pipe = pipeline('OpenGVLab/InternVL2-2B',
backend_config=PytorchEngineConfig(tp=1, device_type='ascend'))
image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
Expand Down
2 changes: 1 addition & 1 deletion docs/zh_cn/get_started/camb/get_started.md
Original file line number Diff line number Diff line change
Expand Up @@ -40,7 +40,7 @@ print(response)

```python
from lmdeploy import pipeline, PytorchEngineConfig
from lmdeploy.vl import load_image
from lmdeploy.multimodal import load_image
pipe = pipeline('OpenGVLab/InternVL2-2B',
backend_config=PytorchEngineConfig(tp=1, device_type='camb'))
image = load_image('https://raw.githubusercontent.com/open-mmlab/mmdeploy/main/tests/data/tiger.jpeg')
Expand Down
Loading
Loading