You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: docs/guides/deploy_local_llm.md
+116-36
Original file line number
Diff line number
Diff line change
@@ -5,71 +5,151 @@ slug: /deploy_local_llm
5
5
6
6
# Deploy a local LLM
7
7
8
-
RAGFlow supports deploying LLMs locally using Ollama or Xinference.
8
+
RAGFlow supports deploying models locally using Ollama or Xinference. If you have locally deployed models to leverage or wish to enable GPU or CUDA for inference acceleration, you can bind Ollama or Xinference into RAGFlow and use either of them as a local "server" for interacting with your local models.
9
9
10
-
## Ollama
10
+
RAGFlow seamlessly integrates with Ollama and Xinference, without the need for further environment configurations. You can use them to deploy two types of local models in RAGFlow: chat models and embedding models.
11
11
12
-
One-click deployment of local LLMs, that is [Ollama](https://github.com/ollama/ollama).
12
+
:::tip NOTE
13
+
This user guide does not intend to cover much of the installation or configuration details of Ollama or Xinference; its focus is on configurations inside RAGFlow. For the most current information, you may need to check out the official site of Ollama or Xinference.
14
+
:::
13
15
14
-
### Install
16
+
##Deploy a local model using Ollama
15
17
16
-
-[Ollama on Linux](https://github.com/ollama/ollama/blob/main/docs/linux.md)
17
-
-[Ollama Windows Preview](https://github.com/ollama/ollama/blob/main/docs/windows.md)
18
-
-[Docker](https://hub.docker.com/r/ollama/ollama)
18
+
[Ollama](https://github.com/ollama/ollama) enables you to run open-source large language models that you deployed locally. It bundles model weights, configurations, and data into a single package, defined by a Modelfile, and optimizes setup and configurations, including GPU usage.
19
19
20
-
### Launch Ollama
20
+
:::note
21
+
- For information about downloading Ollama, see [here](https://github.com/ollama/ollama?tab=readme-ov-file#ollama).
22
+
- For information about configuring Ollama server, see [here](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-configure-ollama-server).
23
+
- For a complete list of supported models and variants, see the [Ollama model library](https://ollama.com/library).
24
+
:::
25
+
26
+
To deploy a local model, e.g., **Llama3**, using Ollama:
27
+
28
+
### 1. Check firewall settings
29
+
30
+
Ensure that your host machine's firewall allows inbound connections on port 11434. For example:
31
+
32
+
```bash
33
+
sudo ufw allow 11434/tcp
34
+
```
35
+
### 2. Ensure Ollama is accessible
36
+
37
+
Restart system and use curl or your web browser to check if the service URL of your Ollama service at `http://localhost:11434` is accessible.
38
+
39
+
```bash
40
+
Ollama is running
41
+
```
42
+
43
+
### 3. Run your local model
44
+
45
+
```bash
46
+
ollama run llama3
47
+
```
48
+
<details>
49
+
<summary>If your Ollama is installed through Docker, run the following instead:</summary>
50
+
51
+
```bash
52
+
docker exec -it ollama ollama run llama3
53
+
```
54
+
</details>
55
+
56
+
### 4. Add Ollama
57
+
58
+
In RAGFlow, click on your logo on the top right of the page **>****Model Providers** and add Ollama to RAGFlow:
In the popup window, complete basic settings for Ollama:
66
+
67
+
1. Because **llama3** is a chat model, choose **chat** as the model type.
68
+
2. Ensure that the model name you enter here *precisely* matches the name of the local model you are running with Ollama.
69
+
3. Ensure that the base URL you enter is accessible to RAGFlow.
70
+
4. OPTIONAL: Switch on the toggle under **Does it support Vision?** if your model includes an image-to-text model.
71
+
72
+
:::caution NOTE
73
+
- If your Ollama and RAGFlow run on the same machine, use `http://localhost:11434` as base URL.
74
+
- If your Ollama and RAGFlow run on the same machine and Ollama is in Docker, use `http://host.docker.internal:11434` as base URL.
75
+
- If your Ollama runs on a different machine from RAGFlow, use `http://<IP_OF_OLLAMA_MACHINE>:11434` as base URL.
76
+
:::
77
+
78
+
:::danger WARNING
79
+
If your Ollama runs on a different machine, you may also need to set the `OLLAMA_HOST` environment variable to `0.0.0.0` in **ollama.service** (Note that this is *NOT* the base URL):
21
80
22
-
Decide which LLM you want to deploy ([here's a list for supported LLM](https://ollama.com/library)), say, **mistral**:
23
81
```bash
24
-
$ ollama run mistral
82
+
Environment="OLLAMA_HOST=0.0.0.0"
25
83
```
26
-
Or,
84
+
85
+
See [this guide](https://github.com/ollama/ollama/blob/main/docs/faq.md#how-do-i-configure-ollama-server) for more information.
86
+
:::
87
+
88
+
:::caution WARNING
89
+
Improper base URL settings will trigger the following error:
27
90
```bash
28
-
$ docker exec -it ollama ollama run mistral
91
+
Max retries exceeded with url: /api/chat (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0xffff98b81ff0>: Failed to establish a new connection: [Errno 111] Connection refused'))
29
92
```
93
+
:::
30
94
31
-
### Use Ollama in RAGFlow
95
+
### 6. Update System Model Settings
32
96
33
-
- Go to 'Settings > Model Providers > Models to be added > Ollama'.
Update your chat model accordingly in **Chat Configuration**:
42
106
43
-
## Xinference
107
+
> If your local model is an embedding model, update it on the configruation page of your knowledge base.
44
108
45
-
Xorbits Inference([Xinference](https://github.com/xorbitsai/inference)) empowers you to unleash the full potential of cutting-edge AI models.
109
+
## Deploy a local model using Xinference
46
110
47
-
### Install
111
+
Xorbits Inference([Xinference](https://github.com/xorbitsai/inference)) enables you to unleash the full potential of cutting-edge AI models.
- For information about installing Xinference Ollama, see [here](https://inference.readthedocs.io/en/latest/getting_started/).
115
+
- For a complete list of supported models, see the [Builtin Models](https://inference.readthedocs.io/en/latest/models/builtin/).
116
+
:::
117
+
118
+
To deploy a local model, e.g., **Llama3**, using Xinference:
119
+
120
+
### 1. Start an Xinference instance
51
121
52
-
To start a local instance of Xinference, run the following command:
53
122
```bash
54
123
$ xinference-local --host 0.0.0.0 --port 9997
55
124
```
56
-
### Launch Xinference
57
125
58
-
Decide which LLM you want to deploy ([here's a list for supported LLM](https://inference.readthedocs.io/en/latest/models/builtin/)), say, **mistral**.
59
-
Execute the following command to launch the model, remember to replace `${quantization}` with your chosen quantization method from the options listed above:
126
+
### 2. Launch your local model
127
+
128
+
Launch your local model (**Mistral**), ensuring that you replace `${quantization}` with your chosen quantization method
> If you have not installed Docker on your local machine (Windows, Mac, or Linux), see [Install Docker Engine](https://docs.docker.com/engine/install/).
27
27
@@ -30,11 +30,11 @@ This quick start guide describes a general process from:
30
30
This section provides instructions on setting up the RAGFlow server on Linux. If you are on a different operating system, no worries. Most steps are alike.
`vm.max_map_count`. This value sets the the maximum number of memory map areas a process may have. Its default value is 65530. While most applications require fewer than a thousand maps, reducing this value can result in abmornal behaviors, and the system will throw out-of-memory errors when a process reaches the limitation.
36
36
37
-
RAGFlow v0.7.0 uses Elasticsearch for multiple recall. Setting the value of `vm.max_map_count` correctly is crucial to the proper functioning the Elasticsearch component.
37
+
RAGFlow v0.7.0 uses Elasticsearch for multiple recall. Setting the value of `vm.max_map_count` correctly is crucial to the proper functioning of the Elasticsearch component.
38
38
39
39
<Tabs
40
40
defaultValue="linux"
@@ -168,7 +168,9 @@ This section provides instructions on setting up the RAGFlow server on Linux. If
168
168
169
169
5. In your web browser, enter the IP address of your server and log in to RAGFlow.
170
170
171
-
> - With default settings, you only need to enter `http://IP_OF_YOUR_MACHINE` (**sans** port number) as the default HTTP serving port `80` can be omitted when using the default configurations.
171
+
:::caution WARNING
172
+
With default settings, you only need to enter `http://IP_OF_YOUR_MACHINE` (**sans** port number) as the default HTTP serving port `80` can be omitted when using the default configurations.
173
+
:::
172
174
173
175
## Configure LLMs
174
176
@@ -188,7 +190,7 @@ To add and configure an LLM:
188
190
189
191
1. Click on your logo on the top right of the page **>****Model Providers**:
> Each RAGFlow account is able to use **text-embedding-v2** for free, a embedding model of Tongyi-Qianwen. This is why you can see Tongyi-Qianwen in the **Added models** list. And you may need to update your Tongyi-Qianwen API key at a later point.
194
196
@@ -286,4 +288,5 @@ Conversations in RAGFlow are based on a particular knowledge base or multiple kn
import {resetWarningCache} from 'prop-types';
Copy file name to clipboardExpand all lines: docs/references/api.md
+4-4
Original file line number
Diff line number
Diff line change
@@ -109,10 +109,10 @@ This method retrieves the history of a specified conversation session.
109
109
- `content_with_weight`: Content of the chunk.
110
110
- `doc_name`: Name of the *hit* document.
111
111
- `img_id`: The image ID of the chunk. It is an optional field only for PDF, PPTX, and images. Call ['GET' /document/get/\<id\>](#get-document-content) to retrieve the image.
112
-
- positions: [page_number, [upleft corner(x, y)], [right bottom(x, y)]], the chunk position, only for PDF.
113
-
- similarity: The hybrid similarity.
114
-
- term_similarity: The keyword simimlarity.
115
-
- vector_similarity: The embedding similarity.
112
+
- `positions`: [page_number, [upleft corner(x, y)], [right bottom(x, y)]], the chunk position, only for PDF.
113
+
- `similarity`: The hybrid similarity.
114
+
- `term_similarity`: The keyword simimlarity.
115
+
- `vector_similarity`: The embedding similarity.
116
116
-`doc_aggs`:
117
117
-`doc_id`: ID of the *hit* document. Call ['GET' /document/get/\<id\>](#get-document-content) to retrieve the document.
0 commit comments