Skip to content

Commit 6f6f27e

Browse files
mpskexmpskex
authored
Merge pull request #1 from myscale/feature/vectorstore_myscale
Feature/vectorstore myscale
2 parents d7942a9 + 9460047 commit 6f6f27e

File tree

8 files changed

+884
-9
lines changed

8 files changed

+884
-9
lines changed

‎docs/ecosystem/myscale.md

+65
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,65 @@
1+
# MyScale
2+
3+
This page covers how to use MyScale vector database within LangChain.
4+
It is broken into two parts: installation and setup, and then references to specific MyScale wrappers.
5+
6+
With MyScale, you can manage both structured and unstructured (vectorized) data, and perform joint queries and analytics on both types of data using SQL. Plus, MyScale's cloud-native OLAP architecture, built on top of ClickHouse, enables lightning-fast data processing even on massive datasets.
7+
8+
## Introduction
9+
10+
[Overview to MyScale and High performance vector search](https://docs.myscale.com/en/overview/)
11+
12+
You can now register on our SaaS and [start a cluster now!](https://docs.myscale.com/en/quickstart/)
13+
14+
If you are also interested in how we managed to integrate SQL and vector, please refer to [this document](https://docs.myscale.com/en/vector-reference/) for further syntax reference.
15+
16+
We also deliver with live demo on huggingface! Please checkout our [huggingface space](https://huggingface.co/myscale)! They search millions of vector within a blink!
17+
18+
## Installation and Setup
19+
- Install the Python SDK with `pip install clickhouse-connect`
20+
21+
### Setting up envrionments
22+
23+
There are two ways to set up parameters for myscale index.
24+
25+
1. Environment Variables
26+
27+
Before you run the app, please set the environment variable with `export`:
28+
`export MYSCALE_URL='<your-endpoints-url>' MYSCALE_PORT=<your-endpoints-port> MYSCALE_USERNAME=<your-username> MYSCALE_PASSWORD=<your-password> ...`
29+
30+
You can easily find your account, password and other info on our SaaS. For details please refer to [this document](https://docs.myscale.com/en/cluster-management/)
31+
Every attributes under `MyScaleSettings` can be set with prefix `MYSCALE_` and is case insensitive.
32+
33+
2. Create `MyScaleSettings` object with parameters
34+
35+
36+
```python
37+
from langchain.vectorstores import MyScale, MyScaleSettings
38+
config = MyScaleSetting(host="<your-backend-url>", port=8443, ...)
39+
index = MyScale(embedding_function, config)
40+
index.add_documents(...)
41+
```
42+
43+
## Wrappers
44+
supported functions:
45+
- `add_texts`
46+
- `add_documents`
47+
- `from_texts`
48+
- `from_documents`
49+
- `similarity_search`
50+
- `asimilarity_search`
51+
- `similarity_search_by_vector`
52+
- `asimilarity_search_by_vector`
53+
- `similarity_search_with_relevance_scores`
54+
55+
### VectorStore
56+
57+
There exists a wrapper around MyScale database, allowing you to use it as a vectorstore,
58+
whether for semantic search or similar example retrieval.
59+
60+
To import this vectorstore:
61+
```python
62+
from langchain.vectorstores import MyScale
63+
```
64+
65+
For a more detailed walkthrough of the MyScale wrapper, see [this notebook](../modules/indexes/vectorstores/examples/myscale.ipynb)

‎docs/modules/indexes/vectorstores/examples/myscale.ipynb

+267
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,267 @@
1+
{
2+
"cells": [
3+
{
4+
"attachments": {},
5+
"cell_type": "markdown",
6+
"id": "683953b3",
7+
"metadata": {},
8+
"source": [
9+
"# MyScale\n",
10+
"\n",
11+
"This notebook shows how to use functionality related to the MyScale vector database."
12+
]
13+
},
14+
{
15+
"cell_type": "code",
16+
"execution_count": 2,
17+
"id": "aac9563e",
18+
"metadata": {},
19+
"outputs": [],
20+
"source": [
21+
"from langchain.embeddings.openai import OpenAIEmbeddings\n",
22+
"from langchain.text_splitter import CharacterTextSplitter\n",
23+
"from langchain.vectorstores import MyScale\n",
24+
"from langchain.document_loaders import TextLoader"
25+
]
26+
},
27+
{
28+
"attachments": {},
29+
"cell_type": "markdown",
30+
"id": "a9d16fa3",
31+
"metadata": {},
32+
"source": [
33+
"## Setting up envrionments\n",
34+
"\n",
35+
"There are two ways to set up parameters for myscale index.\n",
36+
"\n",
37+
"1. Environment Variables\n",
38+
"\n",
39+
" Before you run the app, please set the environment variable with `export`:\n",
40+
" `export MYSCALE_URL='<your-endpoints-url>' MYSCALE_PORT=<your-endpoints-port> MYSCALE_USERNAME=<your-username> MYSCALE_PASSWORD=<your-password> ...`\n",
41+
"\n",
42+
" You can easily find your account, password and other info on our SaaS. For details please refer to [this document](https://docs.myscale.com/en/cluster-management/)\n",
43+
"\n",
44+
" Every attributes under `MyScaleSettings` can be set with prefix `MYSCALE_` and is case insensitive.\n",
45+
"\n",
46+
"2. Create `MyScaleSettings` object with parameters\n",
47+
"\n",
48+
"\n",
49+
" ```python\n",
50+
" from langchain.vectorstores import MyScale, MyScaleSettings\n",
51+
" config = MyScaleSetting(host=\"<your-backend-url>\", port=8443, ...)\n",
52+
" index = MyScale(embedding_function, config)\n",
53+
" index.add_documents(...)\n",
54+
" ```"
55+
]
56+
},
57+
{
58+
"cell_type": "code",
59+
"execution_count": 3,
60+
"id": "a3c3999a",
61+
"metadata": {},
62+
"outputs": [],
63+
"source": [
64+
"from langchain.document_loaders import TextLoader\n",
65+
"loader = TextLoader('../../../state_of_the_union.txt')\n",
66+
"documents = loader.load()\n",
67+
"text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
68+
"docs = text_splitter.split_documents(documents)\n",
69+
"\n",
70+
"embeddings = OpenAIEmbeddings()"
71+
]
72+
},
73+
{
74+
"cell_type": "code",
75+
"execution_count": 4,
76+
"id": "6e104aee",
77+
"metadata": {},
78+
"outputs": [
79+
{
80+
"name": "stderr",
81+
"output_type": "stream",
82+
"text": [
83+
"Inserting data...: 100%|██████████| 42/42 [00:18<00:00, 2.21it/s]\n"
84+
]
85+
}
86+
],
87+
"source": [
88+
"for d in docs:\n",
89+
" d.metadata = {'some': 'metadata'}\n",
90+
"docsearch = MyScale.from_documents(docs, embeddings)\n",
91+
"\n",
92+
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
93+
"docs = docsearch.similarity_search(query)"
94+
]
95+
},
96+
{
97+
"cell_type": "code",
98+
"execution_count": 5,
99+
"id": "9c608226",
100+
"metadata": {},
101+
"outputs": [
102+
{
103+
"name": "stdout",
104+
"output_type": "stream",
105+
"text": [
106+
"As Frances Haugen, who is here with us tonight, has shown, we must hold social media platforms accountable for the national experiment they’re conducting on our children for profit. \n",
107+
"\n",
108+
"It’s time to strengthen privacy protections, ban targeted advertising to children, demand tech companies stop collecting personal data on our children. \n",
109+
"\n",
110+
"And let’s get all Americans the mental health services they need. More people they can turn to for help, and full parity between physical and mental health care. \n",
111+
"\n",
112+
"Third, support our veterans. \n",
113+
"\n",
114+
"Veterans are the best of us. \n",
115+
"\n",
116+
"I’ve always believed that we have a sacred obligation to equip all those we send to war and care for them and their families when they come home. \n",
117+
"\n",
118+
"My administration is providing assistance with job training and housing, and now helping lower-income veterans get VA care debt-free. \n",
119+
"\n",
120+
"Our troops in Iraq and Afghanistan faced many dangers.\n"
121+
]
122+
}
123+
],
124+
"source": [
125+
"print(docs[0].page_content)"
126+
]
127+
},
128+
{
129+
"attachments": {},
130+
"cell_type": "markdown",
131+
"id": "e3a8b105",
132+
"metadata": {},
133+
"source": [
134+
"## Get connection info and data schema"
135+
]
136+
},
137+
{
138+
"cell_type": "code",
139+
"execution_count": null,
140+
"id": "69996818",
141+
"metadata": {},
142+
"outputs": [],
143+
"source": [
144+
"print(str(docsearch))"
145+
]
146+
},
147+
{
148+
"attachments": {},
149+
"cell_type": "markdown",
150+
"id": "f59360c0",
151+
"metadata": {},
152+
"source": [
153+
"## Filtering\n",
154+
"\n",
155+
"You can have direct access to myscale SQL where statement. You can write `WHERE` clause following standard SQL.\n",
156+
"\n",
157+
"**NOTE**: Please be aware of SQL injection, this interface must not be directly called by end-user.\n",
158+
"\n",
159+
"If you custimized your `column_map` under your setting, you search with filter like this:"
160+
]
161+
},
162+
{
163+
"cell_type": "code",
164+
"execution_count": 7,
165+
"id": "232055f6",
166+
"metadata": {},
167+
"outputs": [
168+
{
169+
"name": "stderr",
170+
"output_type": "stream",
171+
"text": [
172+
"Inserting data...: 100%|██████████| 42/42 [00:15<00:00, 2.69it/s]\n"
173+
]
174+
}
175+
],
176+
"source": [
177+
"from langchain.vectorstores import MyScale, MyScaleSettings\n",
178+
"from langchain.document_loaders import TextLoader\n",
179+
"\n",
180+
"loader = TextLoader('../../../state_of_the_union.txt')\n",
181+
"documents = loader.load()\n",
182+
"text_splitter = CharacterTextSplitter(chunk_size=1000, chunk_overlap=0)\n",
183+
"docs = text_splitter.split_documents(documents)\n",
184+
"\n",
185+
"embeddings = OpenAIEmbeddings()\n",
186+
"\n",
187+
"for i, d in enumerate(docs):\n",
188+
" d.metadata = {'doc_id': i}\n",
189+
"\n",
190+
"docsearch = MyScale.from_documents(docs, embeddings)"
191+
]
192+
},
193+
{
194+
"cell_type": "code",
195+
"execution_count": 16,
196+
"id": "ddbcee77",
197+
"metadata": {},
198+
"outputs": [
199+
{
200+
"name": "stdout",
201+
"output_type": "stream",
202+
"text": [
203+
"0.252379834651947 {'doc_id': 6, 'some': ''} And I’m taking robus...\n",
204+
"0.25022566318511963 {'doc_id': 1, 'some': ''} Groups of citizens b...\n",
205+
"0.2469480037689209 {'doc_id': 8, 'some': ''} And so many families...\n",
206+
"0.2428302764892578 {'doc_id': 0, 'some': 'metadata'} As Frances Haugen, w...\n"
207+
]
208+
}
209+
],
210+
"source": [
211+
"meta = docsearch.metadata_column\n",
212+
"output = docsearch.similarity_search_with_relevance_scores('What did the president say about Ketanji Brown Jackson?', \n",
213+
" k=4, where_str=f\"{meta}.doc_id<10\")\n",
214+
"for d, dist in output:\n",
215+
" print(dist, d.metadata, d.page_content[:20] + '...')"
216+
]
217+
},
218+
{
219+
"attachments": {},
220+
"cell_type": "markdown",
221+
"id": "a359ed74",
222+
"metadata": {},
223+
"source": [
224+
"## Deleting your data"
225+
]
226+
},
227+
{
228+
"cell_type": "code",
229+
"execution_count": null,
230+
"id": "fb6a9d36",
231+
"metadata": {},
232+
"outputs": [],
233+
"source": [
234+
"docsearch.drop()"
235+
]
236+
},
237+
{
238+
"cell_type": "code",
239+
"execution_count": null,
240+
"id": "48dbd8e0",
241+
"metadata": {},
242+
"outputs": [],
243+
"source": []
244+
}
245+
],
246+
"metadata": {
247+
"kernelspec": {
248+
"display_name": "Python 3 (ipykernel)",
249+
"language": "python",
250+
"name": "python3"
251+
},
252+
"language_info": {
253+
"codemirror_mode": {
254+
"name": "ipython",
255+
"version": 3
256+
},
257+
"file_extension": ".py",
258+
"mimetype": "text/x-python",
259+
"name": "python",
260+
"nbconvert_exporter": "python",
261+
"pygments_lexer": "ipython3",
262+
"version": "3.8.8"
263+
}
264+
},
265+
"nbformat": 4,
266+
"nbformat_minor": 5
267+
}

‎docs/reference/integrations.md

+2
Original file line numberDiff line numberDiff line change
@@ -45,6 +45,8 @@ The following use cases require specific installs and api keys:
4545
- Set up Elasticsearch backend. If you want to do locally, [this](https://www.elastic.co/guide/en/elasticsearch/reference/7.17/getting-started.html) is a good guide.
4646
- _FAISS_:
4747
- Install requirements with `pip install faiss` for Python 3.7 and `pip install faiss-cpu` for Python 3.10+.
48+
- _MyScale_
49+
- Install requirements with `pip install clickhouse-connect`. For documentations, please refer to [this document](https://docs.myscale.com/en/overview/).
4850
- _Manifest_:
4951
- Install requirements with `pip install manifest-ml` (Note: this is only available in Python 3.8+ currently).
5052
- _OpenSearch_:

‎langchain/vectorstores/__init__.py

+3
Original file line numberDiff line numberDiff line change
@@ -7,6 +7,7 @@
77
from langchain.vectorstores.elastic_vector_search import ElasticVectorSearch
88
from langchain.vectorstores.faiss import FAISS
99
from langchain.vectorstores.milvus import Milvus
10+
from langchain.vectorstores.myscale import MyScale, MyScaleSettings
1011
from langchain.vectorstores.opensearch_vector_search import OpenSearchVectorSearch
1112
from langchain.vectorstores.pinecone import Pinecone
1213
from langchain.vectorstores.qdrant import Qdrant
@@ -26,5 +27,7 @@
2627
"AtlasDB",
2728
"DeepLake",
2829
"Annoy",
30+
"MyScale",
31+
"MyScaleSettings",
2932
"SupabaseVectorStore",
3033
]

0 commit comments

Comments
 (0)