diff --git a/docs/extras/ecosystem/integrations/hologres.mdx b/docs/extras/ecosystem/integrations/hologres.mdx
new file mode 100644
index 0000000000000..66284efbd361b
--- /dev/null
+++ b/docs/extras/ecosystem/integrations/hologres.mdx
@@ -0,0 +1,23 @@
+# Hologres
+
+>[Hologres](https://www.alibabacloud.com/help/en/hologres/latest/introduction) is a unified real-time data warehousing service developed by Alibaba Cloud. You can use Hologres to write, update, process, and analyze large amounts of data in real time.
+>`Hologres` supports standard `SQL` syntax, is compatible with `PostgreSQL`, and supports most PostgreSQL functions. Hologres supports online analytical processing (OLAP) and ad hoc analysis for up to petabytes of data, and provides high-concurrency and low-latency online data services.
+
+>`Hologres` provides **vector database** functionality by adopting [Proxima](https://www.alibabacloud.com/help/en/hologres/latest/vector-processing).
+>`Proxima` is a high-performance software library developed by `Alibaba DAMO Academy`. It allows you to search for the nearest neighbors of vectors. Proxima provides higher stability and performance than similar open source software such as Faiss. Proxima allows you to search for similar text or image embeddings with high throughput and low latency. Hologres is deeply integrated with Proxima to provide a high-performance vector search service.
+
+## Installation and Setup
+
+Click [here](https://www.alibabacloud.com/zh/product/hologres) to fast deploy a Hologres cloud instance.
+
+```bash
+pip install psycopg2
+```
+
+## Vector Store
+
+See a [usage example](/docs/modules/data_connection/vectorstores/integrations/hologres.html).
+
+```python
+from langchain.vectorstores import Hologres
+```
diff --git a/docs/extras/ecosystem/integrations/rockset.mdx b/docs/extras/ecosystem/integrations/rockset.mdx
new file mode 100644
index 0000000000000..6fe71f393c346
--- /dev/null
+++ b/docs/extras/ecosystem/integrations/rockset.mdx
@@ -0,0 +1,19 @@
+# Rockset
+
+>[Rockset](https://rockset.com/product/) is a real-time analytics database service for serving low latency, high concurrency analytical queries at scale. It builds a Converged Index™ on structured and semi-structured data with an efficient store for vector embeddings. Its support for running SQL on schemaless data makes it a perfect choice for running vector search with metadata filters.
+
+## Installation and Setup
+
+Make sure you have Rockset account and go to the web console to get the API key. Details can be found on [the website](https://rockset.com/docs/rest-api/).
+
+```bash
+pip install rockset
+```
+
+## Vector Store
+
+See a [usage example](/docs/modules/data_connection/vectorstores/integrations/rockset.html).
+
+```python
+from langchain.vectorstores import RocksetDB
+```
diff --git a/docs/extras/ecosystem/integrations/singlestoredb.mdx b/docs/extras/ecosystem/integrations/singlestoredb.mdx
new file mode 100644
index 0000000000000..313b7ccbae61d
--- /dev/null
+++ b/docs/extras/ecosystem/integrations/singlestoredb.mdx
@@ -0,0 +1,20 @@
+# SingleStoreDB
+
+>[SingleStoreDB](https://singlestore.com/) is a high-performance distributed SQL database that supports deployment both in the [cloud](https://www.singlestore.com/cloud/) and on-premises. It provides vector storage, and vector functions including [dot_product](https://docs.singlestore.com/managed-service/en/reference/sql-reference/vector-functions/dot_product.html) and [euclidean_distance](https://docs.singlestore.com/managed-service/en/reference/sql-reference/vector-functions/euclidean_distance.html), thereby supporting AI applications that require text similarity matching.
+
+## Installation and Setup
+
+There are several ways to establish a [connection](https://singlestoredb-python.labs.singlestore.com/generated/singlestoredb.connect.html) to the database. You can either set up environment variables or pass named parameters to the `SingleStoreDB constructor`.
+Alternatively, you may provide these parameters to the `from_documents` and `from_texts` methods.
+
+```bash
+pip install singlestoredb
+```
+
+## Vector Store
+
+See a [usage example](/docs/modules/data_connection/vectorstores/integrations/singlestoredb.html).
+
+```python
+from langchain.vectorstores import SingleStoreDB
+```
diff --git a/docs/extras/ecosystem/integrations/sklearn.mdx b/docs/extras/ecosystem/integrations/sklearn.mdx
index cb8723a5b87d2..8f463110c845c 100644
--- a/docs/extras/ecosystem/integrations/sklearn.mdx
+++ b/docs/extras/ecosystem/integrations/sklearn.mdx
@@ -1,15 +1,14 @@
# scikit-learn
-This page covers how to use the scikit-learn package within LangChain.
-It is broken into two parts: installation and setup, and then references to specific scikit-learn wrappers.
+>[scikit-learn](https://scikit-learn.org/stable/) is an open source collection of machine learning algorithms,
+> including some implementations of the [k nearest neighbors](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.NearestNeighbors.html). `SKLearnVectorStore` wraps this implementation and adds the possibility to persist the vector store in json, bson (binary json) or Apache Parquet format.
## Installation and Setup
- Install the Python package with `pip install scikit-learn`
-## Wrappers
-### VectorStore
+## Vector Store
`SKLearnVectorStore` provides a simple wrapper around the nearest neighbor implementation in the
scikit-learn package, allowing you to use it as a vectorstore.
diff --git a/docs/extras/ecosystem/integrations/starrocks.mdx b/docs/extras/ecosystem/integrations/starrocks.mdx
new file mode 100644
index 0000000000000..0c0febacc679b
--- /dev/null
+++ b/docs/extras/ecosystem/integrations/starrocks.mdx
@@ -0,0 +1,21 @@
+# StarRocks
+
+>[StarRocks](https://www.starrocks.io/) is a High-Performance Analytical Database.
+`StarRocks` is a next-gen sub-second MPP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics and ad-hoc query.
+
+>Usually `StarRocks` is categorized into OLAP, and it has showed excellent performance in [ClickBench — a Benchmark For Analytical DBMS](https://benchmark.clickhouse.com/). Since it has a super-fast vectorized execution engine, it could also be used as a fast vectordb.
+
+## Installation and Setup
+
+
+```bash
+pip install pymysql
+```
+
+## Vector Store
+
+See a [usage example](/docs/modules/data_connection/vectorstores/integrations/starrocks.html).
+
+```python
+from langchain.vectorstores import StarRocks
+```
diff --git a/docs/extras/ecosystem/integrations/tigris.mdx b/docs/extras/ecosystem/integrations/tigris.mdx
new file mode 100644
index 0000000000000..7c69141ea4f00
--- /dev/null
+++ b/docs/extras/ecosystem/integrations/tigris.mdx
@@ -0,0 +1,19 @@
+# Tigris
+
+> [Tigris](htttps://tigrisdata.com) is an open source Serverless NoSQL Database and Search Platform designed to simplify building high-performance vector search applications.
+> `Tigris` eliminates the infrastructure complexity of managing, operating, and synchronizing multiple tools, allowing you to focus on building great applications instead.
+
+## Installation and Setup
+
+
+```bash
+pip install tigrisdb openapi-schema-pydantic openai tiktoken
+```
+
+## Vector Store
+
+See a [usage example](/docs/modules/data_connection/vectorstores/integrations/tigris.html).
+
+```python
+from langchain.vectorstores import Tigris
+```
diff --git a/docs/extras/ecosystem/integrations/typesense.mdx b/docs/extras/ecosystem/integrations/typesense.mdx
new file mode 100644
index 0000000000000..d2c64a0a0acae
--- /dev/null
+++ b/docs/extras/ecosystem/integrations/typesense.mdx
@@ -0,0 +1,22 @@
+# Typesense
+
+> [Typesense](https://typesense.org) is an open source, in-memory search engine, that you can either
+> [self-host](https://typesense.org/docs/guide/install-typesense.html#option-2-local-machine-self-hosting) or run
+> on [Typesense Cloud](https://cloud.typesense.org/).
+> `Typesense` focuses on performance by storing the entire index in RAM (with a backup on disk) and also
+> focuses on providing an out-of-the-box developer experience by simplifying available options and setting good defaults.
+
+## Installation and Setup
+
+
+```bash
+pip install typesense openapi-schema-pydantic openai tiktoken
+```
+
+## Vector Store
+
+See a [usage example](/docs/modules/data_connection/vectorstores/integrations/typesense.html).
+
+```python
+from langchain.vectorstores import Typesense
+```
diff --git a/docs/extras/guides/evaluation/comparisons.ipynb b/docs/extras/guides/evaluation/comparisons.ipynb
new file mode 100644
index 0000000000000..28da9942a8387
--- /dev/null
+++ b/docs/extras/guides/evaluation/comparisons.ipynb
@@ -0,0 +1,447 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Comparing Chain Outputs\n",
+ "\n",
+ "Suppose you have two different prompts (or LLMs). How do you know which will generate \"better\" results?\n",
+ "\n",
+ "One automated way to predict the preferred configuration is to use a `PairwiseStringEvaluator` like the `PairwiseStringEvalChain`[[1]](#cite_note-1). This chain prompts an LLM to select which output is preferred, given a specific input.\n",
+ "\n",
+ "For this evalution, we will need 3 things:\n",
+ "1. An evaluator\n",
+ "2. A dataset of inputs\n",
+ "3. 2 (or more) LLMs, Chains, or Agents to compare\n",
+ "\n",
+ "Then we will aggregate the restults to determine the preferred model.\n",
+ "\n",
+ "### Step 1. Create the Evaluator\n",
+ "\n",
+ "In this example, you will use gpt-4 to select which output is preferred."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Optional if you are tracing the notebook\n",
+ "%env LANGCHAIN_PROJECT=\"Comparing Chain Outputs\""
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "from langchain.chat_models import ChatOpenAI\n",
+ "from langchain.evaluation.comparison import PairwiseStringEvalChain\n",
+ "\n",
+ "llm = ChatOpenAI(model=\"gpt-4\")\n",
+ "\n",
+ "eval_chain = PairwiseStringEvalChain.from_llm(llm=llm)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Step 2. Select Dataset\n",
+ "\n",
+ "If you already have real usage data for your LLM, you can use a representative sample. More examples\n",
+ "provide more reliable results. We will use some example queries someone might have about how to use langchain here."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "Found cached dataset parquet (/Users/wfh/.cache/huggingface/datasets/LangChainDatasets___parquet/LangChainDatasets--langchain-howto-queries-bbb748bbee7e77aa/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)\n"
+ ]
+ },
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "d852a1884480457292c90d8bd9d4f1e6",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ " 0%| | 0/1 [00:00\" \n",
+ "llm = ChatOpenAI(temperature=0, model=\"gpt-3.5-turbo-0613\")\n",
+ "\n",
+ "# Initialize the SerpAPIWrapper for search functionality\n",
+ "#Replace in openai_api_key=\"\" with your actual SerpAPI key.\n",
+ "search = SerpAPIWrapper()\n",
+ "\n",
+ "# Define a list of tools offered by the agent\n",
+ "tools = [\n",
+ " Tool(\n",
+ " name=\"Search\",\n",
+ " func=search.run,\n",
+ " coroutine=search.arun,\n",
+ " description=\"Useful when you need to answer questions about current events. You should ask targeted questions.\"\n",
+ " ),\n",
+ "]"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "functions_agent = initialize_agent(tools, llm, agent=AgentType.OPENAI_MULTI_FUNCTIONS, verbose=False)\n",
+ "conversations_agent = initialize_agent(tools, llm, agent=AgentType.CHAT_ZERO_SHOT_REACT_DESCRIPTION, verbose=False)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "list(zip(*[iter(batch_results)]*2)### Step 4. Generate Responses\n",
+ "\n",
+ "We will generate outputs for each of the models before evaluating them."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "data": {
+ "application/vnd.jupyter.widget-view+json": {
+ "model_id": "b076d6bf6680422aa9082d4bad4d98a3",
+ "version_major": 2,
+ "version_minor": 0
+ },
+ "text/plain": [
+ " 0%| | 0/20 [00:00._completion_with_retry in 1.0 seconds as it raised ServiceUnavailableError: The server is overloaded or not ready yet..\n",
+ "Retrying langchain.chat_models.openai.acompletion_with_retry.._completion_with_retry in 1.0 seconds as it raised ServiceUnavailableError: The server is overloaded or not ready yet..\n"
+ ]
+ }
+ ],
+ "source": [
+ "from tqdm.notebook import tqdm\n",
+ "import asyncio\n",
+ "\n",
+ "results = []\n",
+ "agents = [functions_agent, conversations_agent]\n",
+ "concurrency_level = 6 # How many concurrent agents to run. May need to decrease if OpenAI is rate limiting.\n",
+ "\n",
+ "# We will only run the first 20 examples of this dataset to speed things up\n",
+ "# This will lead to larger confidence intervals downstream.\n",
+ "batch = []\n",
+ "for example in tqdm(dataset[:20]):\n",
+ " batch.extend([agent.acall(example['inputs']) for agent in agents])\n",
+ " if len(batch) >= concurrency_level:\n",
+ " batch_results = await asyncio.gather(*batch, return_exceptions=True)\n",
+ " results.extend(list(zip(*[iter(batch_results)]*2)))\n",
+ " batch = []\n",
+ "if batch:\n",
+ " batch_results = await asyncio.gather(*batch, return_exceptions=True)\n",
+ " results.extend(list(zip(*[iter(batch_results)]*2)))"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Step 5. Evaluate Pairs\n",
+ "\n",
+ "Now it's time to evaluate the results. For each agent response, run the evaluation chain to select which output is preferred (or return a tie).\n",
+ "\n",
+ "Randomly select the input order to reduce the likelihood that one model will be preferred just because it is presented first."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "import random\n",
+ "\n",
+ "def predict_preferences(dataset, results) -> list:\n",
+ " preferences = []\n",
+ "\n",
+ " for example, (res_a, res_b) in zip(dataset, results):\n",
+ " input_ = example['inputs']\n",
+ " # Flip a coin to reduce persistent position bias\n",
+ " if random.random() < 0.5:\n",
+ " pred_a, pred_b = res_a, res_b\n",
+ " a, b = \"a\", \"b\"\n",
+ " else:\n",
+ " pred_a, pred_b = res_b, res_a\n",
+ " a, b = \"b\", \"a\"\n",
+ " eval_res = eval_chain.evaluate_string_pairs(\n",
+ " output_a=pred_a['output'] if isinstance(pred_a, dict) else str(pred_a),\n",
+ " output_b=pred_b['output'] if isinstance(pred_b, dict) else str(pred_b),\n",
+ " input=input_\n",
+ " )\n",
+ " if eval_res[\"value\"] == \"A\":\n",
+ " preferences.append(a)\n",
+ " elif eval_res[\"value\"] == \"B\":\n",
+ " preferences.append(b)\n",
+ " else:\n",
+ " preferences.append(None) # No preference\n",
+ " return preferences"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 10,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "preferences = predict_preferences(dataset, results)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {
+ "tags": []
+ },
+ "source": [
+ "**Print out the ratio of preferences.**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 11,
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "OpenAI Functions Agent: 90.00%\n",
+ "Structured Chat Agent: 10.00%\n"
+ ]
+ }
+ ],
+ "source": [
+ "from collections import Counter\n",
+ "\n",
+ "name_map = {\n",
+ " \"a\": \"OpenAI Functions Agent\",\n",
+ " \"b\": \"Structured Chat Agent\",\n",
+ "}\n",
+ "counts = Counter(preferences)\n",
+ "pref_ratios = {\n",
+ " k: v/len(preferences) for k, v in\n",
+ " counts.items()\n",
+ "}\n",
+ "for k, v in pref_ratios.items():\n",
+ " print(f\"{name_map.get(k)}: {v:.2%}\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "### Estimate Confidence Intervals\n",
+ "\n",
+ "The results seem pretty clear, but if you want to have a better sense of how confident we are, that model \"A\" (the OpenAI Functions Agent) is the preferred model, we can calculate confidence intervals. \n",
+ "\n",
+ "Below, use the Wilson score to estimate the confidence interval."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "from math import sqrt\n",
+ "\n",
+ "def wilson_score_interval(preferences: list, which: str = \"a\", z: float = 1.96) -> tuple:\n",
+ " \"\"\"Estimate the confidence interval using the Wilson score.\n",
+ " \n",
+ " See: https://en.wikipedia.org/wiki/Binomial_proportion_confidence_interval#Wilson_score_interval\n",
+ " for more details, including when to use it and when it should not be used.\n",
+ " \"\"\"\n",
+ " total_preferences = preferences.count('a') + preferences.count('b')\n",
+ " n_s = preferences.count(which)\n",
+ "\n",
+ " if total_preferences == 0:\n",
+ " return (0, 0)\n",
+ "\n",
+ " p_hat = n_s / total_preferences\n",
+ "\n",
+ " denominator = 1 + (z**2) / total_preferences\n",
+ " adjustment = (z / denominator) * sqrt(p_hat*(1-p_hat)/total_preferences + (z**2)/(4*total_preferences*total_preferences))\n",
+ " center = (p_hat + (z**2) / (2*total_preferences)) / denominator\n",
+ " lower_bound = min(max(center - adjustment, 0.0), 1.0)\n",
+ " upper_bound = min(max(center + adjustment, 0.0), 1.0)\n",
+ "\n",
+ " return (lower_bound, upper_bound)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 13,
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "The \"OpenAI Functions Agent\" would be preferred between 69.90% and 97.21% percent of the time (with 95% confidence).\n",
+ "The \"Structured Chat Agent\" would be preferred between 2.79% and 30.10% percent of the time (with 95% confidence).\n"
+ ]
+ }
+ ],
+ "source": [
+ "for which_, name in name_map.items():\n",
+ " low, high = wilson_score_interval(preferences, which=which_)\n",
+ " print(f'The \"{name}\" would be preferred between {low:.2%} and {high:.2%} percent of the time (with 95% confidence).')"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Print out the p-value.**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "The p-value is 0.00040. If the null hypothesis is true (i.e., if the selected eval chain actually has no preference between the models),\n",
+ "then there is a 0.04025% chance of observing the OpenAI Functions Agent be preferred at least 18\n",
+ "times out of 20 trials.\n"
+ ]
+ }
+ ],
+ "source": [
+ "from scipy import stats\n",
+ "preferred_model = max(pref_ratios, key=pref_ratios.get)\n",
+ "successes = preferences.count(preferred_model)\n",
+ "n = len(preferences) - preferences.count(None)\n",
+ "p_value = stats.binom_test(successes, n, p=0.5, alternative='two-sided')\n",
+ "print(f\"\"\"The p-value is {p_value:.5f}. If the null hypothesis is true (i.e., if the selected eval chain actually has no preference between the models),\n",
+ "then there is a {p_value:.5%} chance of observing the {name_map.get(preferred_model)} be preferred at least {successes}\n",
+ "times out of {n} trials.\"\"\")"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "_1. Note: Automated evals are still an open research topic and are best used alongside other evaluation approaches. \n",
+ "LLM preferences exhibit biases, including banal ones like the order of outputs.\n",
+ "In choosing preferences, \"ground truth\" may not be taken into account, which may lead to scores that aren't grounded in utility._"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/docs/extras/guides/evaluation/criteria_eval_chain.ipynb b/docs/extras/guides/evaluation/criteria_eval_chain.ipynb
new file mode 100644
index 0000000000000..b754bc71e2f88
--- /dev/null
+++ b/docs/extras/guides/evaluation/criteria_eval_chain.ipynb
@@ -0,0 +1,264 @@
+{
+ "cells": [
+ {
+ "cell_type": "markdown",
+ "id": "4cf569a7-9a1d-4489-934e-50e57760c907",
+ "metadata": {},
+ "source": [
+ "# Evaluating Custom Criteria\n",
+ "\n",
+ "Suppose you want to test a model's output against a custom rubric or custom set of criteria, how would you go about testing this?\n",
+ "\n",
+ "The `CriteriaEvalChain` is a convenient way to predict whether an LLM or Chain's output complies with a set of criteria, so long as you can\n",
+ "describe those criteria in regular language. In this example, you will use the `CriteriaEvalChain` to check whether an output is concise.\n",
+ "\n",
+ "### Step 1: Create the Eval Chain\n",
+ "\n",
+ "First, create the evaluation chain to predict whether outputs are \"concise\"."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "6005ebe8-551e-47a5-b4df-80575a068552",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "from langchain.chat_models import ChatOpenAI\n",
+ "from langchain.evaluation.criteria import CriteriaEvalChain\n",
+ "\n",
+ "llm = ChatOpenAI(temperature=0)\n",
+ "criterion = \"conciseness\"\n",
+ "eval_chain = CriteriaEvalChain.from_llm(llm=llm, criteria=criterion)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "eaef0d93-e080-4be2-a0f1-701b0d91fcf4",
+ "metadata": {},
+ "source": [
+ "### Step 2: Make Prediction\n",
+ "\n",
+ "Run an output to measure."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 2,
+ "id": "68b1a348-cf41-40bf-9667-e79683464cf2",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "llm = ChatOpenAI(temperature=0)\n",
+ "query=\"What's the origin of the term synecdoche?\"\n",
+ "prediction = llm.predict(query)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "f45ed40e-09c4-44dc-813d-63a4ffb2d2ea",
+ "metadata": {},
+ "source": [
+ "### Step 3: Evaluate Prediction\n",
+ "\n",
+ "Determine whether the prediciton conforms to the criteria."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "id": "22f83fb8-82f4-4310-a877-68aaa0789199",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'reasoning': '1. Conciseness: The submission is concise and to the point. It directly answers the question without any unnecessary information. Therefore, the submission meets the criterion of conciseness.\\n\\nY', 'value': 'Y', 'score': 1}\n"
+ ]
+ }
+ ],
+ "source": [
+ "eval_result = eval_chain.evaluate_strings(prediction=prediction, input=query)\n",
+ "print(eval_result)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "id": "8c4ec9dd-6557-4f23-8480-c822eb6ec552",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "['conciseness',\n",
+ " 'relevance',\n",
+ " 'coherence',\n",
+ " 'harmfulness',\n",
+ " 'maliciousness',\n",
+ " 'helpfulness',\n",
+ " 'controversiality',\n",
+ " 'mysogyny',\n",
+ " 'criminality',\n",
+ " 'insensitive']"
+ ]
+ },
+ "execution_count": 4,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# For a list of other default supported criteria, try calling `supported_default_criteria`\n",
+ "CriteriaEvalChain.get_supported_default_criteria()"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "2eb7dedb-913a-4d9e-b48a-9521425d1008",
+ "metadata": {},
+ "source": [
+ "## Multiple Criteria\n",
+ "\n",
+ "To check whether an output complies with all of a list of default criteria, pass in a list! Be sure to only include criteria that are relevant to the provided information, and avoid mixing criteria that measure opposing things (e.g., harmfulness and helpfulness)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "id": "50c067f7-bc6e-4d6c-ba34-97a72023be27",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'reasoning': 'Conciseness: The submission is not concise and does not answer the given task. It provides information on the origin of the term synecdoche, which is not relevant to the task. Therefore, the submission does not meet the criterion of conciseness.\\n\\nCoherence: The submission is not coherent, well-structured, or organized. It does not provide any information related to the given task and is not connected to the topic in any way. Therefore, the submission does not meet the criterion of coherence.\\n\\nConclusion: The submission does not meet all criteria.', 'value': 'N', 'score': 0}\n"
+ ]
+ }
+ ],
+ "source": [
+ "criteria = [\"conciseness\", \"coherence\"]\n",
+ "eval_chain = CriteriaEvalChain.from_llm(llm=llm, criteria=criteria)\n",
+ "eval_result = eval_chain.evaluate_strings(prediction=prediction, input=query)\n",
+ "print(eval_result)"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "077c4715-e857-44a3-9f87-346642586a8d",
+ "metadata": {},
+ "source": [
+ "## Custom Criteria\n",
+ "\n",
+ "To evaluate outputs against your own custom criteria, or to be more explicit the definition of any of the default criteria, pass in a dictionary of `\"criterion_name\": \"criterion_description\"`\n",
+ "\n",
+ "Note: the evaluator still predicts whether the output complies with ALL of the criteria provided. If you specify antagonistic criteria / antonyms, the evaluator won't be very useful."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "id": "bafa0a11-2617-4663-84bf-24df7d0736be",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'reasoning': '1. Criteria: numeric: Does the output contain numeric information?\\n- The submission does not contain any numeric information.\\n- Conclusion: The submission meets the criteria.', 'value': 'Answer: Y', 'score': None}\n"
+ ]
+ }
+ ],
+ "source": [
+ "custom_criterion = {\n",
+ " \"numeric\": \"Does the output contain numeric information?\"\n",
+ "}\n",
+ "\n",
+ "eval_chain = CriteriaEvalChain.from_llm(llm=llm, criteria=custom_criterion)\n",
+ "eval_result = eval_chain.evaluate_strings(prediction=prediction, input=query)\n",
+ "print(eval_result)"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "id": "6db12a16-0058-4a14-8064-8528540963d8",
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'reasoning': '- complements-user: The submission directly answers the question asked and provides additional information about the population of Lagos. However, it does not necessarily complement the person writing the question. \\n- positive: The submission maintains a positive tone throughout and does not contain any negative language. \\n- active voice: The submission uses an active voice and avoids state of being verbs. \\n\\nTherefore, the submission meets all criteria. \\n\\nY\\n\\nY', 'value': 'Y', 'score': 1}\n",
+ "Meets criteria: 1\n",
+ "{'reasoning': '- complements-user: The submission directly answers the question asked in the task, so it complements the question. Therefore, the answer meets this criterion. \\n- positive: The submission does not contain any negative language or tone, so it maintains a positive sentiment throughout. Therefore, the answer meets this criterion. \\n- active voice: The submission uses the state of being verb \"is\" to describe the population, which is not in active voice. Therefore, the answer does not meet this criterion. \\n\\nAnswer: N', 'value': 'N', 'score': 0}\n",
+ "Does not meet criteria: 0\n"
+ ]
+ }
+ ],
+ "source": [
+ "# You can specify multiple criteria in the dictionary. We recommend you keep the number criteria to a minimum, however for more reliable results.\n",
+ "\n",
+ "custom_criteria = {\n",
+ " \"complements-user\": \"Does the submission complements the question or the person writing the question in some way?\",\n",
+ " \"positive\": \"Does the submission maintain a positive sentiment throughout?\",\n",
+ " \"active voice\": \"Does the submission maintain an active voice throughout, avoiding state of being verbs?\",\n",
+ "}\n",
+ "\n",
+ "eval_chain = CriteriaEvalChain.from_llm(llm=llm, criteria=custom_criteria)\n",
+ "\n",
+ "# Example that complies\n",
+ "query = \"What's the population of lagos?\"\n",
+ "eval_result = eval_chain.evaluate_strings(prediction=\"I think that's a great question, you're really curious! About 30 million people live in Lagos, Nigeria, as of 2023.\", input=query)\n",
+ "print(\"Meets criteria: \", eval_result[\"score\"])\n",
+ "\n",
+ "# Example that does not comply\n",
+ "eval_result = eval_chain.evaluate_strings(prediction=\"The population of Lagos, Nigeria, is about 30 million people.\", input=query)\n",
+ "print(\"Does not meet criteria: \", eval_result[\"score\"])"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "99e3c242-5b12-4bd5-b487-64990a159655",
+ "metadata": {},
+ "outputs": [],
+ "source": []
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 5
+}
diff --git a/docs/extras/guides/evaluation/generic_agent_evaluation.ipynb b/docs/extras/guides/evaluation/generic_agent_evaluation.ipynb
index 85a71e3e9a83c..c56cca9a9e1c8 100644
--- a/docs/extras/guides/evaluation/generic_agent_evaluation.ipynb
+++ b/docs/extras/guides/evaluation/generic_agent_evaluation.ipynb
@@ -4,9 +4,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "# Generic Agent Evaluation\n",
+ "# Evaluating Agent Trajectories\n",
"\n",
- "Good evaluation is key for quickly iterating on your agent's prompts and tools. Here we provide an example of how to use the TrajectoryEvalChain to evaluate your agent."
+ "Good evaluation is key for quickly iterating on your agent's prompts and tools. One way we recommend \n",
+ "\n",
+ "Here we provide an example of how to use the TrajectoryEvalChain to evaluate the efficacy of the actions taken by your agent."
]
},
{
@@ -21,7 +23,9 @@
{
"cell_type": "code",
"execution_count": 2,
- "metadata": {},
+ "metadata": {
+ "tags": []
+ },
"outputs": [],
"source": [
"from langchain import Wikipedia\n",
@@ -39,7 +43,7 @@
"\n",
"math_llm = OpenAI(temperature=0)\n",
"\n",
- "llm_math_chain = LLMMathChain(llm=math_llm, verbose=True)\n",
+ "llm_math_chain = LLMMathChain.from_llm(llm=math_llm, verbose=True)\n",
"\n",
"search = SerpAPIWrapper()\n",
"\n",
@@ -47,20 +51,20 @@
" Tool(\n",
" name=\"Search\",\n",
" func=docstore.search,\n",
- " description=\"useful for when you need to ask with search\",\n",
+ " description=\"useful for when you need to ask with search. Must call before lookup.\",\n",
" ),\n",
" Tool(\n",
" name=\"Lookup\",\n",
" func=docstore.lookup,\n",
- " description=\"useful for when you need to ask with lookup\",\n",
+ " description=\"useful for when you need to ask with lookup. Only call after a successfull 'Search'.\",\n",
" ),\n",
" Tool(\n",
" name=\"Calculator\",\n",
" func=llm_math_chain.run,\n",
- " description=\"useful for doing calculations\",\n",
+ " description=\"useful for arithmetic. Expects strict numeric input, no words.\",\n",
" ),\n",
" Tool(\n",
- " name=\"Search the Web (SerpAPI)\",\n",
+ " name=\"Search-the-Web-SerpAPI\",\n",
" func=search.run,\n",
" description=\"useful for when you need to answer questions about current events\",\n",
" ),\n",
@@ -70,12 +74,12 @@
" memory_key=\"chat_history\", return_messages=True, output_key=\"output\"\n",
")\n",
"\n",
- "llm = ChatOpenAI(temperature=0, model_name=\"gpt-3.5-turbo\")\n",
+ "llm = ChatOpenAI(temperature=0, model_name=\"gpt-3.5-turbo-0613\")\n",
"\n",
"agent = initialize_agent(\n",
" tools,\n",
" llm,\n",
- " agent=AgentType.CHAT_CONVERSATIONAL_REACT_DESCRIPTION,\n",
+ " agent=AgentType.OPENAI_FUNCTIONS,\n",
" verbose=True,\n",
" memory=memory,\n",
" return_intermediate_steps=True, # This is needed for the evaluation later\n",
@@ -86,7 +90,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Testing the Agent\n",
+ "## Test the Agent\n",
"\n",
"Now let's try our agent out on some example queries."
]
@@ -94,7 +98,9 @@
{
"cell_type": "code",
"execution_count": 3,
- "metadata": {},
+ "metadata": {
+ "tags": []
+ },
"outputs": [
{
"name": "stdout",
@@ -102,16 +108,22 @@
"text": [
"\n",
"\n",
- "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
- "\u001b[32;1m\u001b[1;3m{\n",
- " \"action\": \"Search the Web (SerpAPI)\",\n",
- " \"action_input\": \"How many ping pong balls would it take to fill the entire Empire State Building?\"\n",
- "}\u001b[0m\n",
- "Observation: \u001b[31;1m\u001b[1;3m12.8 billion. The volume of the Empire State Building Googles in at around 37 million ft³. A golf ball comes in at about 2.5 in³.\u001b[0m\n",
- "Thought:\u001b[32;1m\u001b[1;3m{\n",
- " \"action\": \"Final Answer\",\n",
- " \"action_input\": \"It would take approximately 12.8 billion ping pong balls to fill the entire Empire State Building.\"\n",
- "}\u001b[0m\n",
+ "\u001b[1m> Entering new chain...\u001b[0m\n",
+ "\u001b[32;1m\u001b[1;3m\n",
+ "Invoking: `Calculator` with `1040000 / (4/100)^3 / 1000000`\n",
+ "responded: {content}\n",
+ "\n",
+ "\u001b[0m\n",
+ "\n",
+ "\u001b[1m> Entering new chain...\u001b[0m\n",
+ "1040000 / (4/100)^3 / 1000000\u001b[32;1m\u001b[1;3m```text\n",
+ "1040000 / (4/100)**3 / 1000000\n",
+ "```\n",
+ "...numexpr.evaluate(\"1040000 / (4/100)**3 / 1000000\")...\n",
+ "\u001b[0m\n",
+ "Answer: \u001b[33;1m\u001b[1;3m16249.999999999998\u001b[0m\n",
+ "\u001b[1m> Finished chain.\u001b[0m\n",
+ "\u001b[38;5;200m\u001b[1;3mAnswer: 16249.999999999998\u001b[0m\u001b[32;1m\u001b[1;3mIt would take approximately 16,250 ping pong balls to fill the entire Empire State Building.\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n"
]
@@ -129,13 +141,15 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "This looks good! Let's try it out on another query."
+ "This looks alright.. Let's try it out on another query."
]
},
{
"cell_type": "code",
"execution_count": 4,
- "metadata": {},
+ "metadata": {
+ "tags": []
+ },
"outputs": [
{
"name": "stdout",
@@ -143,43 +157,49 @@
"text": [
"\n",
"\n",
- "\u001b[1m> Entering new AgentExecutor chain...\u001b[0m\n",
- "\u001b[32;1m\u001b[1;3m{\n",
- " \"action\": \"Calculator\",\n",
- " \"action_input\": \"The length of the Eiffel Tower is 324 meters. The distance from coast to coast in the US is approximately 4,828 kilometers. First, we need to convert 4,828 kilometers to meters, which gives us 4,828,000 meters. To find out how many Eiffel Towers we need, we can divide 4,828,000 by 324. This gives us approximately 14,876 Eiffel Towers.\"\n",
- "}\u001b[0m\n",
+ "\u001b[1m> Entering new chain...\u001b[0m\n",
+ "\u001b[32;1m\u001b[1;3m\n",
+ "Invoking: `Search` with `length of the US from coast to coast`\n",
+ "\n",
+ "\n",
+ "\u001b[0m\u001b[36;1m\u001b[1;3m\n",
+ "== Watercraft ==\u001b[0m\u001b[32;1m\u001b[1;3m\n",
+ "Invoking: `Search` with `distance from coast to coast of the US`\n",
+ "\n",
+ "\n",
+ "\u001b[0m\u001b[36;1m\u001b[1;3mThe Oregon Coast is a coastal region of the U.S. state of Oregon. It is bordered by the Pacific Ocean to its west and the Oregon Coast Range to the east, and stretches approximately 362 miles (583 km) from the California state border in the south to the Columbia River in the north. The region is not a specific geological, environmental, or political entity, and includes the Columbia River Estuary.\n",
+ "The Oregon Beach Bill of 1967 allows free beach access to everyone. In return for a pedestrian easement and relief from construction, the bill eliminates property taxes on private beach land and allows its owners to retain certain beach land rights.Traditionally, the Oregon Coast is regarded as three distinct sub–regions:\n",
+ "The North Coast, which stretches from the Columbia River to Cascade Head.\n",
+ "The Central Coast, which stretches from Cascade Head to Reedsport.\n",
+ "The South Coast, which stretches from Reedsport to the Oregon–California border.The largest city is Coos Bay, population 16,700 in Coos County on the South Coast. U.S. Route 101 is the primary highway from Brookings to Astoria and is known for its scenic overlooks of the Pacific Ocean. Over 80 state parks and recreation areas dot the Oregon Coast. However, only a few highways cross the Coast Range to the interior: US 30, US 26, OR 6, US 20, OR 18, OR 34, OR 126, OR 38, and OR 42. OR 18 and US 20 are considered among the dangerous roads in the state.The Oregon Coast includes Clatsop County, Tillamook County, Lincoln County, western Lane County, western Douglas County, Coos County, and Curry County.\u001b[0m\u001b[32;1m\u001b[1;3m\n",
+ "Invoking: `Calculator` with `362 miles * 5280 feet`\n",
"\n",
- "\u001b[1m> Entering new LLMMathChain chain...\u001b[0m\n",
- "The length of the Eiffel Tower is 324 meters. The distance from coast to coast in the US is approximately 4,828 kilometers. First, we need to convert 4,828 kilometers to meters, which gives us 4,828,000 meters. To find out how many Eiffel Towers we need, we can divide 4,828,000 by 324. This gives us approximately 14,876 Eiffel Towers.\u001b[32;1m\u001b[1;3m\n",
- "```text\n",
- "4828000 / 324\n",
+ "\n",
+ "\u001b[0m\n",
+ "\n",
+ "\u001b[1m> Entering new chain...\u001b[0m\n",
+ "362 miles * 5280 feet\u001b[32;1m\u001b[1;3m```text\n",
+ "362 * 5280\n",
"```\n",
- "...numexpr.evaluate(\"4828000 / 324\")...\n",
+ "...numexpr.evaluate(\"362 * 5280\")...\n",
"\u001b[0m\n",
- "Answer: \u001b[33;1m\u001b[1;3m14901.234567901234\u001b[0m\n",
+ "Answer: \u001b[33;1m\u001b[1;3m1911360\u001b[0m\n",
"\u001b[1m> Finished chain.\u001b[0m\n",
+ "\u001b[38;5;200m\u001b[1;3mAnswer: 1911360\u001b[0m\u001b[32;1m\u001b[1;3m\n",
+ "Invoking: `Calculator` with `1911360 feet / 1063 feet`\n",
"\n",
- "Observation: \u001b[38;5;200m\u001b[1;3mAnswer: 14901.234567901234\u001b[0m\n",
- "Thought:\u001b[32;1m\u001b[1;3m{\n",
- " \"action\": \"Calculator\",\n",
- " \"action_input\": \"The length of the Eiffel Tower is 324 meters. The distance from coast to coast in the US is approximately 4,828 kilometers. First, we need to convert 4,828 kilometers to meters, which gives us 4,828,000 meters. To find out how many Eiffel Towers we need, we can divide 4,828,000 by 324. This gives us approximately 14,901 Eiffel Towers.\"\n",
- "}\u001b[0m\n",
"\n",
- "\u001b[1m> Entering new LLMMathChain chain...\u001b[0m\n",
- "The length of the Eiffel Tower is 324 meters. The distance from coast to coast in the US is approximately 4,828 kilometers. First, we need to convert 4,828 kilometers to meters, which gives us 4,828,000 meters. To find out how many Eiffel Towers we need, we can divide 4,828,000 by 324. This gives us approximately 14,901 Eiffel Towers.\u001b[32;1m\u001b[1;3m\n",
- "```text\n",
- "4828000 / 324\n",
+ "\u001b[0m\n",
+ "\n",
+ "\u001b[1m> Entering new chain...\u001b[0m\n",
+ "1911360 feet / 1063 feet\u001b[32;1m\u001b[1;3m```text\n",
+ "1911360 / 1063\n",
"```\n",
- "...numexpr.evaluate(\"4828000 / 324\")...\n",
+ "...numexpr.evaluate(\"1911360 / 1063\")...\n",
"\u001b[0m\n",
- "Answer: \u001b[33;1m\u001b[1;3m14901.234567901234\u001b[0m\n",
+ "Answer: \u001b[33;1m\u001b[1;3m1798.0809031044214\u001b[0m\n",
"\u001b[1m> Finished chain.\u001b[0m\n",
- "\n",
- "Observation: \u001b[38;5;200m\u001b[1;3mAnswer: 14901.234567901234\u001b[0m\n",
- "Thought:\u001b[32;1m\u001b[1;3m{\n",
- " \"action\": \"Final Answer\",\n",
- " \"action_input\": \"If you laid the Eiffel Tower end to end, you would need approximately 14,901 Eiffel Towers to cover the US from coast to coast.\"\n",
- "}\u001b[0m\n",
+ "\u001b[38;5;200m\u001b[1;3mAnswer: 1798.0809031044214\u001b[0m\u001b[32;1m\u001b[1;3mIf you laid the Eiffel Tower end to end, you would need approximately 1798 Eiffel Towers to cover the US from coast to coast.\u001b[0m\n",
"\n",
"\u001b[1m> Finished chain.\u001b[0m\n"
]
@@ -205,16 +225,17 @@
{
"cell_type": "code",
"execution_count": 5,
- "metadata": {},
+ "metadata": {
+ "tags": []
+ },
"outputs": [],
"source": [
"from langchain.evaluation.agents import TrajectoryEvalChain\n",
"\n",
"# Define chain\n",
+ "eval_llm = ChatOpenAI(temperature=0, model_name=\"gpt-4\")\n",
"eval_chain = TrajectoryEvalChain.from_llm(\n",
- " llm=ChatOpenAI(\n",
- " temperature=0, model_name=\"gpt-4\"\n",
- " ), # Note: This must be a ChatOpenAI model\n",
+ " llm=eval_llm, # Note: This must be a chat model\n",
" agent_tools=agent.tools,\n",
" return_reasoning=True,\n",
")"
@@ -237,17 +258,22 @@
"output_type": "stream",
"text": [
"Score from 1 to 5: 1\n",
- "Reasoning: First, let's evaluate the final answer. The final answer is incorrect because it uses the volume of golf balls instead of ping pong balls. The answer is not helpful.\n",
+ "Reasoning: i. Is the final answer helpful?\n",
+ "The final answer is not helpful because it is incorrect. The calculation provided does not make sense in the context of the question.\n",
"\n",
- "Second, does the model use a logical sequence of tools to answer the question? The model only used one tool, which was the Search the Web (SerpAPI). It did not use the Calculator tool to calculate the correct volume of ping pong balls.\n",
+ "ii. Does the AI language use a logical sequence of tools to answer the question?\n",
+ "The AI language model does not use a logical sequence of tools. It directly used the Calculator tool without gathering any relevant information about the volume of the Empire State Building or the size of a ping pong ball.\n",
"\n",
- "Third, does the AI language model use the tools in a helpful way? The model used the Search the Web (SerpAPI) tool, but the output was not helpful because it provided information about golf balls instead of ping pong balls.\n",
+ "iii. Does the AI language model use the tools in a helpful way?\n",
+ "The AI language model does not use the tools in a helpful way. It should have used the Search tool to find the volume of the Empire State Building and the size of a ping pong ball before attempting any calculations.\n",
"\n",
- "Fourth, does the AI language model use too many steps to answer the question? The model used only one step, which is not too many. However, it should have used more steps to provide a correct answer.\n",
+ "iv. Does the AI language model use too many steps to answer the question?\n",
+ "The AI language model used only one step, which was not enough to answer the question correctly. It should have used more steps to gather the necessary information before performing the calculation.\n",
"\n",
- "Fifth, are the appropriate tools used to answer the question? The model should have used the Search tool to find the volume of the Empire State Building and the volume of a ping pong ball. Then, it should have used the Calculator tool to calculate the number of ping pong balls needed to fill the building.\n",
+ "v. Are the appropriate tools used to answer the question?\n",
+ "The appropriate tools were not used to answer the question. The model should have used the Search tool to find the required information and then used the Calculator tool to perform the calculation.\n",
"\n",
- "Judgment: Given the incorrect final answer and the inappropriate use of tools, we give the model a score of 1.\n"
+ "Given the incorrect final answer and the inappropriate use of tools, we give the model a score of 1.\n"
]
}
],
@@ -258,12 +284,10 @@
" test_outputs_one[\"output\"],\n",
")\n",
"\n",
- "evaluation = eval_chain(\n",
- " inputs={\n",
- " \"question\": question,\n",
- " \"answer\": answer,\n",
- " \"agent_trajectory\": eval_chain.get_agent_trajectory(steps),\n",
- " },\n",
+ "evaluation = eval_chain.evaluate_agent_trajectory(\n",
+ " input=test_outputs_one[\"input\"],\n",
+ " output=test_outputs_one[\"output\"],\n",
+ " agent_trajectory=test_outputs_one[\"intermediate_steps\"],\n",
")\n",
"\n",
"print(\"Score from 1 to 5: \", evaluation[\"score\"])\n",
@@ -274,51 +298,97 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "That seems about right. Let's try the second query."
+ "**That seems about right. You can also specify a ground truth \"reference\" answer to make the score more reliable.**"
]
},
{
"cell_type": "code",
- "execution_count": 7,
- "metadata": {},
+ "execution_count": 13,
+ "metadata": {
+ "tags": []
+ },
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
- "Score from 1 to 5: 3\n",
+ "Score from 1 to 5: 1\n",
"Reasoning: i. Is the final answer helpful?\n",
- "Yes, the final answer is helpful as it provides an approximate number of Eiffel Towers needed to cover the US from coast to coast.\n",
+ "The final answer is not helpful, as it is incorrect. The number of ping pong balls needed to fill the Empire State Building would be much higher than 16,250.\n",
"\n",
"ii. Does the AI language use a logical sequence of tools to answer the question?\n",
- "No, the AI language model does not use a logical sequence of tools. It directly uses the Calculator tool without first using the Search or Lookup tools to find the necessary information (length of the Eiffel Tower and distance from coast to coast in the US).\n",
+ "The AI language model does not use a logical sequence of tools. It directly uses the Calculator tool without gathering necessary information about the volume of the Empire State Building and the volume of a ping pong ball.\n",
"\n",
"iii. Does the AI language model use the tools in a helpful way?\n",
- "The AI language model uses the Calculator tool in a helpful way to perform the calculation, but it should have used the Search or Lookup tools first to find the required information.\n",
+ "The AI language model does not use the tools in a helpful way. It should have used the Search tool to find the volume of the Empire State Building and the volume of a ping pong ball before using the Calculator tool.\n",
"\n",
"iv. Does the AI language model use too many steps to answer the question?\n",
- "No, the AI language model does not use too many steps. However, it repeats the same step twice, which is unnecessary.\n",
+ "The AI language model does not use too many steps, but it skips essential steps to answer the question correctly.\n",
"\n",
"v. Are the appropriate tools used to answer the question?\n",
- "Not entirely. The AI language model should have used the Search or Lookup tools to find the required information before using the Calculator tool.\n",
+ "The appropriate tools are not used to answer the question. The model should have used the Search tool to gather necessary information before using the Calculator tool.\n",
"\n",
- "Given the above evaluation, the AI language model's performance can be scored as follows:\n"
+ "Given the incorrect final answer and the inappropriate use of tools, we give the model a score of 1.\n"
]
}
],
"source": [
- "question, steps, answer = (\n",
- " test_outputs_two[\"input\"],\n",
- " test_outputs_two[\"intermediate_steps\"],\n",
- " test_outputs_two[\"output\"],\n",
+ "evaluation = eval_chain.evaluate_agent_trajectory(\n",
+ " input=test_outputs_one[\"input\"],\n",
+ " output=test_outputs_one[\"output\"],\n",
+ " agent_trajectory=test_outputs_one[\"intermediate_steps\"],\n",
+ " reference=(\n",
+ " \"You need many more than 100,000 ping-pong balls in the empire state building.\"\n",
+ " )\n",
")\n",
+ " \n",
"\n",
- "evaluation = eval_chain(\n",
- " inputs={\n",
- " \"question\": question,\n",
- " \"answer\": answer,\n",
- " \"agent_trajectory\": eval_chain.get_agent_trajectory(steps),\n",
- " },\n",
+ "print(\"Score from 1 to 5: \", evaluation[\"score\"])\n",
+ "print(\"Reasoning: \", evaluation[\"reasoning\"])"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "**Let's try the second query. This time, use the async API. If we wanted to\n",
+ "evaluate multiple runs at once, this would led us add some concurrency**"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 14,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "Score from 1 to 5: 2\n",
+ "Reasoning: i. Is the final answer helpful?\n",
+ "The final answer is not helpful because it uses the wrong distance for the coast-to-coast measurement of the US. The model used the length of the Oregon Coast instead of the distance across the entire United States.\n",
+ "\n",
+ "ii. Does the AI language use a logical sequence of tools to answer the question?\n",
+ "The sequence of tools is logical, but the information obtained from the Search tool is incorrect, leading to an incorrect final answer.\n",
+ "\n",
+ "iii. Does the AI language model use the tools in a helpful way?\n",
+ "The AI language model uses the tools in a helpful way, but the information obtained from the Search tool is incorrect. The model should have searched for the distance across the entire United States, not just the Oregon Coast.\n",
+ "\n",
+ "iv. Does the AI language model use too many steps to answer the question?\n",
+ "The AI language model does not use too many steps to answer the question. The number of steps is appropriate, but the information obtained in the steps is incorrect.\n",
+ "\n",
+ "v. Are the appropriate tools used to answer the question?\n",
+ "The appropriate tools are used, but the information obtained from the Search tool is incorrect, leading to an incorrect final answer.\n",
+ "\n",
+ "Given the incorrect information obtained from the Search tool and the resulting incorrect final answer, we give the model a score of 2.\n"
+ ]
+ }
+ ],
+ "source": [
+ "evaluation = await eval_chain.aevaluate_agent_trajectory(\n",
+ " input=test_outputs_two[\"input\"],\n",
+ " output=test_outputs_two[\"output\"],\n",
+ " agent_trajectory=test_outputs_two[\"intermediate_steps\"],\n",
")\n",
"\n",
"print(\"Score from 1 to 5: \", evaluation[\"score\"])\n",
@@ -329,7 +399,11 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "That also sounds about right. In conclusion, the TrajectoryEvalChain allows us to use GPT-4 to score both our agent's outputs and tool use in addition to giving us the reasoning behind the evaluation."
+ "## Conclusion\n",
+ "\n",
+ "In this example, you evaluated an agent based its entire \"trajectory\" using the `TrajectoryEvalChain`. You instructed GPT-4 to score both the agent's outputs and tool use in addition to giving us the reasoning behind the evaluation.\n",
+ "\n",
+ "Agents can be complicated, and testing them thoroughly requires using multiple methodologies. Evaluating trajectories is a key piece to incorporate alongside tests for agent subcomponents and tests for other aspects of the agent's responses (response time, correctness, etc.) "
]
}
],
@@ -349,7 +423,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.9.1"
+ "version": "3.11.3"
},
"vscode": {
"interpreter": {
@@ -358,5 +432,5 @@
}
},
"nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
}
diff --git a/docs/extras/modules/agents/toolkits/office365.ipynb b/docs/extras/modules/agents/toolkits/office365.ipynb
new file mode 100644
index 0000000000000..07c4e14f0bd5f
--- /dev/null
+++ b/docs/extras/modules/agents/toolkits/office365.ipynb
@@ -0,0 +1,238 @@
+{
+ "cells": [
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "# Office365 Toolkit\n",
+ "\n",
+ "This notebook walks through connecting LangChain to Office365 email and calendar.\n",
+ "\n",
+ "To use this toolkit, you will need to set up your credentials explained in the [Microsoft Graph authentication and authorization overview](https://learn.microsoft.com/en-us/graph/auth/). Once you've received a CLIENT_ID and CLIENT_SECRET, you can input them as environmental variables below."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "!pip install --upgrade O365 > /dev/null\n",
+ "!pip install beautifulsoup4 > /dev/null # This is optional but is useful for parsing HTML messages"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Assign Environmental Variables\n",
+ "\n",
+ "The toolkit will read the CLIENT_ID and CLIENT_SECRET environmental variables to authenticate the user so you need to set them here. You will also need to set your OPENAI_API_KEY to use the agent later."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "# Set environmental variables here"
+ ]
+ },
+ {
+ "attachments": {},
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Create the Toolkit and Get Tools\n",
+ "\n",
+ "To start, you need to create the toolkit, so you can access its tools later."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 3,
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "[O365SearchEvents(name='events_search', description=\" Use this tool to search for the user's calendar events. The input must be the start and end datetimes for the search query. The output is a JSON list of all the events in the user's calendar between the start and end times. You can assume that the user can not schedule any meeting over existing meetings, and that the user is busy during meetings. Any times without events are free for the user. \", args_schema=, return_direct=False, verbose=False, callbacks=None, callback_manager=None, handle_tool_error=False, account=Account Client Id: f32a022c-3c4c-4d10-a9d8-f6a9a9055302),\n",
+ " O365CreateDraftMessage(name='create_email_draft', description='Use this tool to create a draft email with the provided message fields.', args_schema=, return_direct=False, verbose=False, callbacks=None, callback_manager=None, handle_tool_error=False, account=Account Client Id: f32a022c-3c4c-4d10-a9d8-f6a9a9055302),\n",
+ " O365SearchEmails(name='messages_search', description='Use this tool to search for email messages. The input must be a valid Microsoft Graph v1.0 $search query. The output is a JSON list of the requested resource.', args_schema=, return_direct=False, verbose=False, callbacks=None, callback_manager=None, handle_tool_error=False, account=Account Client Id: f32a022c-3c4c-4d10-a9d8-f6a9a9055302),\n",
+ " O365SendEvent(name='send_event', description='Use this tool to create and send an event with the provided event fields.', args_schema=, return_direct=False, verbose=False, callbacks=None, callback_manager=None, handle_tool_error=False, account=Account Client Id: f32a022c-3c4c-4d10-a9d8-f6a9a9055302),\n",
+ " O365SendMessage(name='send_email', description='Use this tool to send an email with the provided message fields.', args_schema=, return_direct=False, verbose=False, callbacks=None, callback_manager=None, handle_tool_error=False, account=Account Client Id: f32a022c-3c4c-4d10-a9d8-f6a9a9055302)]"
+ ]
+ },
+ "execution_count": 3,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "from langchain.agents.agent_toolkits import O365Toolkit\n",
+ "\n",
+ "toolkit = O365Toolkit()\n",
+ "tools = toolkit.get_tools()\n",
+ "tools"
+ ]
+ },
+ {
+ "cell_type": "markdown",
+ "metadata": {},
+ "source": [
+ "## Use within an Agent"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 4,
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "from langchain import OpenAI\n",
+ "from langchain.agents import initialize_agent, AgentType"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 5,
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [],
+ "source": [
+ "llm = OpenAI(temperature=0)\n",
+ "agent = initialize_agent(\n",
+ " tools=toolkit.get_tools(),\n",
+ " llm=llm,\n",
+ " verbose=False,\n",
+ " agent=AgentType.STRUCTURED_CHAT_ZERO_SHOT_REACT_DESCRIPTION,\n",
+ ")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 6,
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "'The draft email was created correctly.'"
+ ]
+ },
+ "execution_count": 6,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "agent.run(\"Create an email draft for me to edit of a letter from the perspective of a sentient parrot\"\n",
+ " \" who is looking to collaborate on some research with her\"\n",
+ " \" estranged friend, a cat. Under no circumstances may you send the message, however.\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 12,
+ "metadata": {
+ "tags": []
+ },
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "\"I found one draft in your drafts folder about collaboration. It was sent on 2023-06-16T18:22:17+0000 and the subject was 'Collaboration Request'.\""
+ ]
+ },
+ "execution_count": 12,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "agent.run(\"Could you search in my drafts folder and let me know if any of them are about collaboration?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 8,
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stderr",
+ "output_type": "stream",
+ "text": [
+ "/home/vscode/langchain-py-env/lib/python3.11/site-packages/O365/utils/windows_tz.py:639: PytzUsageWarning: The zone attribute is specific to pytz's interface; please migrate to a new time zone provider. For more details on how to do so, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html\n",
+ " iana_tz.zone if isinstance(iana_tz, tzinfo) else iana_tz)\n",
+ "/home/vscode/langchain-py-env/lib/python3.11/site-packages/O365/utils/utils.py:463: PytzUsageWarning: The zone attribute is specific to pytz's interface; please migrate to a new time zone provider. For more details on how to do so, see https://pytz-deprecation-shim.readthedocs.io/en/latest/migration.html\n",
+ " timezone = date_time.tzinfo.zone if date_time.tzinfo is not None else None\n"
+ ]
+ },
+ {
+ "data": {
+ "text/plain": [
+ "'I have scheduled a meeting with a sentient parrot to discuss research collaborations on October 3, 2023 at 2 pm Easter Time. Please let me know if you need to make any changes.'"
+ ]
+ },
+ "execution_count": 8,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "agent.run(\"Can you schedule a 30 minute meeting with a sentient parrot to discuss research collaborations on October 3, 2023 at 2 pm Easter Time?\")"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 9,
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "\"Yes, you have an event on October 3, 2023 with a sentient parrot. The event is titled 'Meeting with sentient parrot' and is scheduled from 6:00 PM to 6:30 PM.\""
+ ]
+ },
+ "execution_count": 9,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "agent.run(\"Can you tell me if I have any events on October 3, 2023 in Eastern Time, and if so, tell me if any of them are with a sentient parrot?\")"
+ ]
+ }
+ ],
+ "metadata": {
+ "kernelspec": {
+ "display_name": "Python 3 (ipykernel)",
+ "language": "python",
+ "name": "python3"
+ },
+ "language_info": {
+ "codemirror_mode": {
+ "name": "ipython",
+ "version": 3
+ },
+ "file_extension": ".py",
+ "mimetype": "text/x-python",
+ "name": "python",
+ "nbconvert_exporter": "python",
+ "pygments_lexer": "ipython3",
+ "version": "3.11.3"
+ }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 4
+}
diff --git a/docs/extras/modules/data_connection/document_loaders/integrations/recursive_url_loader.ipynb b/docs/extras/modules/data_connection/document_loaders/integrations/recursive_url_loader.ipynb
index 2d402184ac953..b35b814af4b56 100644
--- a/docs/extras/modules/data_connection/document_loaders/integrations/recursive_url_loader.ipynb
+++ b/docs/extras/modules/data_connection/document_loaders/integrations/recursive_url_loader.ipynb
@@ -1,6 +1,7 @@
{
"cells": [
{
+ "attachments": {},
"cell_type": "markdown",
"id": "5a7cc773",
"metadata": {},
@@ -17,7 +18,7 @@
"\n",
"But, the challenge is traversing the tree of child pages and actually assembling that list!\n",
" \n",
- "We do this using the `RecusiveUrlLoader`.\n",
+ "We do this using the `RecursiveUrlLoader`.\n",
"\n",
"This also gives us the flexibility to exclude some children (e.g., the `api` directory with > 800 child pages)."
]
@@ -29,10 +30,11 @@
"metadata": {},
"outputs": [],
"source": [
- "from langchain.document_loaders.recursive_url_loader import RecusiveUrlLoader"
+ "from langchain.document_loaders.recursive_url_loader import RecursiveUrlLoader"
]
},
{
+ "attachments": {},
"cell_type": "markdown",
"id": "6384c057",
"metadata": {},
@@ -48,7 +50,7 @@
"outputs": [],
"source": [
"url = 'https://js.langchain.com/docs/modules/memory/examples/'\n",
- "loader=RecusiveUrlLoader(url=url)\n",
+ "loader=RecursiveUrlLoader(url=url)\n",
"docs=loader.load()"
]
},
@@ -119,6 +121,7 @@
]
},
{
+ "attachments": {},
"cell_type": "markdown",
"id": "40fc13ef",
"metadata": {},
@@ -137,7 +140,7 @@
"source": [
"url = 'https://js.langchain.com/docs/'\n",
"exclude_dirs=['https://js.langchain.com/docs/api/']\n",
- "loader=RecusiveUrlLoader(url=url,exclude_dirs=exclude_dirs)\n",
+ "loader=RecursiveUrlLoader(url=url,exclude_dirs=exclude_dirs)\n",
"docs=loader.load()"
]
},
diff --git a/docs/extras/modules/data_connection/vectorstores/integrations/alibabacloud_opensearch.ipynb b/docs/extras/modules/data_connection/vectorstores/integrations/alibabacloud_opensearch.ipynb
index 2a31d7f9d8768..9be50011575c2 100644
--- a/docs/extras/modules/data_connection/vectorstores/integrations/alibabacloud_opensearch.ipynb
+++ b/docs/extras/modules/data_connection/vectorstores/integrations/alibabacloud_opensearch.ipynb
@@ -2,28 +2,34 @@
"cells": [
{
"cell_type": "markdown",
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"source": [
"# Alibaba Cloud OpenSearch\n",
"\n",
- ">[Alibaba Cloud Opensearch](https://www.alibabacloud.com/product/opensearch) OpenSearch is a one-stop platform to develop intelligent search services. OpenSearch was built based on the large-scale distributed search engine developed by Alibaba. OpenSearch serves more than 500 business cases in Alibaba Group and thousands of Alibaba Cloud customers. OpenSearch helps develop search services in different search scenarios, including e-commerce, O2O, multimedia, the content industry, communities and forums, and big data query in enterprises.\n",
+ ">[Alibaba Cloud Opensearch](https://www.alibabacloud.com/product/opensearch) is a one-stop platform to develop intelligent search services. `OpenSearch` was built on the large-scale distributed search engine developed by `Alibaba`. `OpenSearch` serves more than 500 business cases in Alibaba Group and thousands of Alibaba Cloud customers. `OpenSearch` helps develop search services in different search scenarios, including e-commerce, O2O, multimedia, the content industry, communities and forums, and big data query in enterprises.\n",
"\n",
- ">OpenSearch helps you develop high quality, maintenance-free, and high performance intelligent search services to provide your users with high search efficiency and accuracy.\n",
+ ">`OpenSearch` helps you develop high quality, maintenance-free, and high performance intelligent search services to provide your users with high search efficiency and accuracy.\n",
"\n",
- ">OpenSearch provides the vector search feature. In specific scenarios, especially test question search and image search scenarios, you can use the vector search feature together with the multimodal search feature to improve the accuracy of search results. This topic describes the syntax and usage notes of vector indexes.\n",
+ ">`OpenSearch` provides the vector search feature. In specific scenarios, especially test question search and image search scenarios, you can use the vector search feature together with the multimodal search feature to improve the accuracy of search results. This topic describes the syntax and usage notes of vector indexes.\n",
"\n",
"This notebook shows how to use functionality related to the `Alibaba Cloud OpenSearch Vector Search Edition`.\n",
"To run, you should have an [OpenSearch Vector Search Edition](https://opensearch.console.aliyun.com) instance up and running:\n",
- "- Read the [help document](https://www.alibabacloud.com/help/en/opensearch/latest/vector-search) to quickly familiarize and configure OpenSearch Vector Search Edition instance.\n"
+ "\n",
+ "Read the [help document](https://www.alibabacloud.com/help/en/opensearch/latest/vector-search) to quickly familiarize and configure OpenSearch Vector Search Edition instance.\n"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#!pip install alibabacloud-ha3engine"
]
},
{
"cell_type": "markdown",
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"source": [
"After completing the configuration, follow these steps to connect to the instance, index documents, and perform vector retrieval."
]
@@ -33,6 +39,9 @@
"execution_count": null,
"metadata": {
"collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ },
"pycharm": {
"name": "#%%\n"
}
@@ -49,9 +58,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"source": [
"Split documents and get embeddings by call OpenAI API"
]
@@ -61,6 +68,9 @@
"execution_count": null,
"metadata": {
"collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ },
"pycharm": {
"name": "#%%\n"
}
@@ -80,7 +90,6 @@
{
"cell_type": "markdown",
"metadata": {
- "collapsed": false,
"pycharm": {
"name": "#%% md\n"
}
@@ -94,6 +103,9 @@
"execution_count": null,
"metadata": {
"collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ },
"pycharm": {
"name": "#%%\n"
}
@@ -133,9 +145,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"source": [
"Create an opensearch access instance by settings."
]
@@ -145,6 +155,9 @@
"execution_count": null,
"metadata": {
"collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ },
"pycharm": {
"name": "#%%\n"
}
@@ -159,9 +172,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"source": [
"or"
]
@@ -171,6 +182,9 @@
"execution_count": null,
"metadata": {
"collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ },
"pycharm": {
"name": "#%%\n"
}
@@ -183,9 +197,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"source": [
"Add texts and build index."
]
@@ -195,6 +207,9 @@
"execution_count": null,
"metadata": {
"collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ },
"pycharm": {
"name": "#%%\n"
}
@@ -208,9 +223,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"source": [
"Query and retrieve data."
]
@@ -220,6 +233,9 @@
"execution_count": null,
"metadata": {
"collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ },
"pycharm": {
"name": "#%%\n"
}
@@ -233,9 +249,7 @@
},
{
"cell_type": "markdown",
- "metadata": {
- "collapsed": false
- },
+ "metadata": {},
"source": [
"Query and retrieve data with metadata\n"
]
@@ -245,6 +259,9 @@
"execution_count": null,
"metadata": {
"collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ },
"pycharm": {
"name": "#%%\n"
}
@@ -260,7 +277,6 @@
{
"cell_type": "markdown",
"metadata": {
- "collapsed": false,
"pycharm": {
"name": "#%% md\n"
}
@@ -272,23 +288,23 @@
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3",
+ "display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
- "version": 2
+ "version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
- "pygments_lexer": "ipython2",
- "version": "2.7.6"
+ "pygments_lexer": "ipython3",
+ "version": "3.10.6"
}
},
"nbformat": 4,
- "nbformat_minor": 0
+ "nbformat_minor": 4
}
diff --git a/docs/extras/modules/data_connection/vectorstores/integrations/awadb.ipynb b/docs/extras/modules/data_connection/vectorstores/integrations/awadb.ipynb
index aedfc8feb127f..93bf1a6d9750d 100644
--- a/docs/extras/modules/data_connection/vectorstores/integrations/awadb.ipynb
+++ b/docs/extras/modules/data_connection/vectorstores/integrations/awadb.ipynb
@@ -6,8 +6,9 @@
"metadata": {},
"source": [
"# AwaDB\n",
- "[AwaDB](https://github.com/awa-ai/awadb) is an AI Native database for the search and storage of embedding vectors used by LLM Applications.\n",
- "This notebook shows how to use functionality related to the AwaDB."
+ ">[AwaDB](https://github.com/awa-ai/awadb) is an AI Native database for the search and storage of embedding vectors used by LLM Applications.\n",
+ "\n",
+ "This notebook shows how to use functionality related to the `AwaDB`."
]
},
{
@@ -184,7 +185,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.11.1"
+ "version": "3.10.6"
}
},
"nbformat": 4,
diff --git a/docs/extras/modules/data_connection/vectorstores/integrations/azuresearch.ipynb b/docs/extras/modules/data_connection/vectorstores/integrations/azuresearch.ipynb
index cf0ee7d0eab9a..c36f525fd2ab7 100644
--- a/docs/extras/modules/data_connection/vectorstores/integrations/azuresearch.ipynb
+++ b/docs/extras/modules/data_connection/vectorstores/integrations/azuresearch.ipynb
@@ -1,19 +1,19 @@
{
"cells": [
{
- "attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
- "# Azure Cognitive Search"
+ "# Azure Cognitive Search\n",
+ "\n",
+ ">[Azure Cognitive Search](https://learn.microsoft.com/en-us/azure/search/search-what-is-azure-search) (formerly known as `Azure Search`) is a cloud search service that gives developers infrastructure, APIs, and tools for building a rich search experience over private, heterogeneous content in web, mobile, and enterprise applications.\n"
]
},
{
- "attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
- "# Install Azure Cognitive Search SDK"
+ "## Install Azure Cognitive Search SDK"
]
},
{
@@ -27,7 +27,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
@@ -49,7 +48,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
@@ -74,7 +72,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
@@ -95,7 +92,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
@@ -120,7 +116,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
@@ -148,7 +143,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
@@ -187,7 +181,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
@@ -226,7 +219,7 @@
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3.9.13 ('.venv': venv)",
+ "display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
@@ -240,9 +233,8 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.11.3"
+ "version": "3.10.6"
},
- "orig_nbformat": 4,
"vscode": {
"interpreter": {
"hash": "645053d6307d413a1a75681b5ebb6449bb2babba4bcb0bf65a1ddc3dbefb108a"
@@ -250,5 +242,5 @@
}
},
"nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
}
diff --git a/docs/extras/modules/data_connection/vectorstores/integrations/chroma.ipynb b/docs/extras/modules/data_connection/vectorstores/integrations/chroma.ipynb
index d4f6944b63014..631b0f045e396 100644
--- a/docs/extras/modules/data_connection/vectorstores/integrations/chroma.ipynb
+++ b/docs/extras/modules/data_connection/vectorstores/integrations/chroma.ipynb
@@ -9,20 +9,6 @@
"\n",
">[Chroma](https://docs.trychroma.com/getting-started) is a AI-native open-source vector database focused on developer productivity and happiness. Chroma is licensed under Apache 2.0.\n",
"\n",
- "\n",
- " ![\"Discord\"](\"https://img.shields.io/discord/1073293645303795742\")
\n",
- " \n",
- "\n",
- " ![\"License\"](\"https://img.shields.io/static/v1?label=license&message=Apache)
\n",
- " \n",
- "![\"Integration](\"https://github.com/chroma-core/chroma/actions/workflows/chroma-integration-test.yml/badge.svg?branch=main\")
\n",
- "\n",
- "- [Website](https://www.trychroma.com/)\n",
- "- [Documentation](https://docs.trychroma.com/)\n",
- "- [Twitter](https://twitter.com/trychroma)\n",
- "- [Discord](https://discord.gg/MMeYNTmh3x)\n",
- "\n",
- "Chroma is fully-typed, fully-tested and fully-documented.\n",
"\n",
"Install Chroma with:\n",
"\n",
@@ -47,19 +33,6 @@
"View full docs at [docs](https://docs.trychroma.com/reference/Collection). To access these methods directly, you can do `._collection_.method()`\n"
]
},
- {
- "cell_type": "code",
- "execution_count": null,
- "id": "12e83df7",
- "metadata": {},
- "outputs": [],
- "source": [
- "# first install dependencies\n",
- "!pip install langchain\n",
- "!pip install langchainplus_sdk\n",
- "!pip install chromadb\n"
- ]
- },
{
"cell_type": "markdown",
"id": "2b5ffbf8",
@@ -491,6 +464,73 @@
"source": [
"retriever.get_relevant_documents(query)[0]"
]
+ },
+ {
+ "cell_type": "markdown",
+ "id": "275dbd0a",
+ "metadata": {},
+ "source": [
+ "### Filtering on metadata\n",
+ "\n",
+ "It can be helpful to narrow down the collection before working with it.\n",
+ "\n",
+ "For example, collections can be filtered on metadata using the get method."
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 17,
+ "id": "a5119221",
+ "metadata": {},
+ "outputs": [
+ {
+ "name": "stdout",
+ "output_type": "stream",
+ "text": [
+ "{'source': 'some_other_source'}\n",
+ "{'ids': ['1'], 'embeddings': None, 'documents': ['Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \\n\\nTonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \\n\\nOne of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \\n\\nAnd I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.'], 'metadatas': [{'source': 'some_other_source'}]}\n"
+ ]
+ }
+ ],
+ "source": [
+ "# create simple ids\n",
+ "ids = [str(i) for i in range(1, len(docs) + 1)]\n",
+ "\n",
+ "# add data\n",
+ "example_db = Chroma.from_documents(docs, embedding_function, ids=ids)\n",
+ "docs = example_db.similarity_search(query)\n",
+ "print(docs[0].metadata)\n",
+ "\n",
+ "# update the source for a document\n",
+ "docs[0].metadata = {\"source\": \"some_other_source\"}\n",
+ "example_db.update_document(ids[0], docs[0])\n",
+ "print(example_db._collection.get(ids=[ids[0]]))"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 18,
+ "id": "81600dc1",
+ "metadata": {},
+ "outputs": [
+ {
+ "data": {
+ "text/plain": [
+ "{'ids': ['1'],\n",
+ " 'embeddings': None,\n",
+ " 'documents': ['Tonight. I call on the Senate to: Pass the Freedom to Vote Act. Pass the John Lewis Voting Rights Act. And while you’re at it, pass the Disclose Act so Americans can know who is funding our elections. \\n\\nTonight, I’d like to honor someone who has dedicated his life to serve this country: Justice Stephen Breyer—an Army veteran, Constitutional scholar, and retiring Justice of the United States Supreme Court. Justice Breyer, thank you for your service. \\n\\nOne of the most serious constitutional responsibilities a President has is nominating someone to serve on the United States Supreme Court. \\n\\nAnd I did that 4 days ago, when I nominated Circuit Court of Appeals Judge Ketanji Brown Jackson. One of our nation’s top legal minds, who will continue Justice Breyer’s legacy of excellence.'],\n",
+ " 'metadatas': [{'source': 'some_other_source'}]}"
+ ]
+ },
+ "execution_count": 18,
+ "metadata": {},
+ "output_type": "execute_result"
+ }
+ ],
+ "source": [
+ "# filter collection for updated source\n",
+ "example_db.get(where={\"source\": \"some_other_source\"})"
+ ]
}
],
"metadata": {
@@ -509,7 +549,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.11.3"
+ "version": "3.10.6"
}
},
"nbformat": 4,
diff --git a/docs/extras/modules/data_connection/vectorstores/integrations/elasticsearch.ipynb b/docs/extras/modules/data_connection/vectorstores/integrations/elasticsearch.ipynb
index ac1c65b3aef3b..188b9cd24020c 100644
--- a/docs/extras/modules/data_connection/vectorstores/integrations/elasticsearch.ipynb
+++ b/docs/extras/modules/data_connection/vectorstores/integrations/elasticsearch.ipynb
@@ -14,22 +14,12 @@
"This notebook shows how to use functionality related to the `Elasticsearch` database."
]
},
- {
- "cell_type": "markdown",
- "source": [
- "# ElasticVectorSearch class"
- ],
- "metadata": {
- "id": "tKSYjyTBtSLc"
- },
- "id": "tKSYjyTBtSLc"
- },
{
"cell_type": "markdown",
"id": "b66c12b2-2a07-4136-ac77-ce1c9fa7a409",
"metadata": {
- "tags": [],
- "id": "b66c12b2-2a07-4136-ac77-ce1c9fa7a409"
+ "id": "b66c12b2-2a07-4136-ac77-ce1c9fa7a409",
+ "tags": []
},
"source": [
"## Installation"
@@ -104,8 +94,8 @@
"execution_count": null,
"id": "d6197931-cbe5-460c-a5e6-b5eedb83887c",
"metadata": {
- "tags": [],
- "id": "d6197931-cbe5-460c-a5e6-b5eedb83887c"
+ "id": "d6197931-cbe5-460c-a5e6-b5eedb83887c",
+ "tags": []
},
"outputs": [],
"source": [
@@ -117,9 +107,9 @@
"execution_count": null,
"id": "67ab8afa-f7c6-4fbf-b596-cb512da949da",
"metadata": {
- "tags": [],
"id": "67ab8afa-f7c6-4fbf-b596-cb512da949da",
- "outputId": "fd16b37f-cb76-40a9-b83f-eab58dd0d912"
+ "outputId": "fd16b37f-cb76-40a9-b83f-eab58dd0d912",
+ "tags": []
},
"outputs": [
{
@@ -141,8 +131,8 @@
"cell_type": "markdown",
"id": "f6030187-0bd7-4798-8372-a265036af5e0",
"metadata": {
- "tags": [],
- "id": "f6030187-0bd7-4798-8372-a265036af5e0"
+ "id": "f6030187-0bd7-4798-8372-a265036af5e0",
+ "tags": []
},
"source": [
"## Example"
@@ -153,8 +143,8 @@
"execution_count": null,
"id": "aac9563e",
"metadata": {
- "tags": [],
- "id": "aac9563e"
+ "id": "aac9563e",
+ "tags": []
},
"outputs": [],
"source": [
@@ -169,8 +159,8 @@
"execution_count": null,
"id": "a3c3999a",
"metadata": {
- "tags": [],
- "id": "a3c3999a"
+ "id": "a3c3999a",
+ "tags": []
},
"outputs": [],
"source": [
@@ -189,8 +179,8 @@
"execution_count": null,
"id": "12eb86d8",
"metadata": {
- "tags": [],
- "id": "12eb86d8"
+ "id": "12eb86d8",
+ "tags": []
},
"outputs": [],
"source": [
@@ -235,43 +225,49 @@
},
{
"cell_type": "markdown",
- "source": [
- "# ElasticKnnSearch Class\n",
- "The `ElasticKnnSearch` implements features allowing storing vectors and documents in Elasticsearch for use with approximate [kNN search](https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html)"
- ],
+ "id": "FheGPztJsrRB",
"metadata": {
"id": "FheGPztJsrRB"
},
- "id": "FheGPztJsrRB"
+ "source": [
+ "# ElasticKnnSearch Class\n",
+ "The `ElasticKnnSearch` implements features allowing storing vectors and documents in Elasticsearch for use with approximate [kNN search](https://www.elastic.co/guide/en/elasticsearch/reference/current/knn-search.html)"
+ ]
},
{
"cell_type": "code",
- "source": [
- "!pip install langchain elasticsearch"
- ],
+ "execution_count": null,
+ "id": "gRVcbh5zqCJQ",
"metadata": {
"id": "gRVcbh5zqCJQ"
},
- "execution_count": null,
"outputs": [],
- "id": "gRVcbh5zqCJQ"
+ "source": [
+ "!pip install langchain elasticsearch"
+ ]
},
{
"cell_type": "code",
- "source": [
- "from langchain.vectorstores.elastic_vector_search import ElasticKnnSearch\n",
- "from langchain.embeddings import ElasticsearchEmbeddings\n",
- "import elasticsearch"
- ],
+ "execution_count": null,
+ "id": "TJtqiw5AqBp8",
"metadata": {
"id": "TJtqiw5AqBp8"
},
- "execution_count": null,
"outputs": [],
- "id": "TJtqiw5AqBp8"
+ "source": [
+ "from langchain.vectorstores.elastic_vector_search import ElasticKnnSearch\n",
+ "from langchain.embeddings import ElasticsearchEmbeddings\n",
+ "import elasticsearch"
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "id": "XHfC0As6qN3T",
+ "metadata": {
+ "id": "XHfC0As6qN3T"
+ },
+ "outputs": [],
"source": [
"# Initialize ElasticsearchEmbeddings\n",
"model_id = \"\"\n",
@@ -281,16 +277,16 @@
"es_password = \"es_pass\"\n",
"test_index = \"\"\n",
"# input_field = \"your_input_field\" # if different from 'text_field'"
- ],
- "metadata": {
- "id": "XHfC0As6qN3T"
- },
- "execution_count": null,
- "outputs": [],
- "id": "XHfC0As6qN3T"
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "id": "UkTipx1lqc3h",
+ "metadata": {
+ "id": "UkTipx1lqc3h"
+ },
+ "outputs": [],
"source": [
"# Generate embedding object\n",
"embeddings = ElasticsearchEmbeddings.from_credentials(\n",
@@ -300,16 +296,16 @@
" es_user=es_user,\n",
" es_password=es_password,\n",
")"
- ],
- "metadata": {
- "id": "UkTipx1lqc3h"
- },
- "execution_count": null,
- "outputs": [],
- "id": "UkTipx1lqc3h"
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "id": "74psgD0oqjYK",
+ "metadata": {
+ "id": "74psgD0oqjYK"
+ },
+ "outputs": [],
"source": [
"# Initialize ElasticKnnSearch\n",
"knn_search = ElasticKnnSearch(\n",
@@ -319,26 +315,26 @@
" index_name=test_index,\n",
" embedding=embeddings,\n",
")"
- ],
- "metadata": {
- "id": "74psgD0oqjYK"
- },
- "execution_count": null,
- "outputs": [],
- "id": "74psgD0oqjYK"
+ ]
},
{
"cell_type": "markdown",
- "source": [
- "## Test adding vectors"
- ],
+ "id": "7AfgIKLWqnQl",
"metadata": {
"id": "7AfgIKLWqnQl"
},
- "id": "7AfgIKLWqnQl"
+ "source": [
+ "## Test adding vectors"
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "id": "yNUUIaL9qmze",
+ "metadata": {
+ "id": "yNUUIaL9qmze"
+ },
+ "outputs": [],
"source": [
"# Test `add_texts` method\n",
"texts = [\"Hello, world!\", \"Machine learning is fun.\", \"I love Python.\"]\n",
@@ -351,26 +347,26 @@
" \"Python is great for data analysis.\",\n",
"]\n",
"knn_search.from_texts(new_texts, dims=dims)"
- ],
- "metadata": {
- "id": "yNUUIaL9qmze"
- },
- "execution_count": null,
- "outputs": [],
- "id": "yNUUIaL9qmze"
+ ]
},
{
"cell_type": "markdown",
- "source": [
- "## Test knn search using query vector builder "
- ],
+ "id": "0zdR-Iubquov",
"metadata": {
"id": "0zdR-Iubquov"
},
- "id": "0zdR-Iubquov"
+ "source": [
+ "## Test knn search using query vector builder "
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "id": "bwR4jYvqqxTo",
+ "metadata": {
+ "id": "bwR4jYvqqxTo"
+ },
+ "outputs": [],
"source": [
"# Test `knn_search` method with model_id and query_text\n",
"query = \"Hello\"\n",
@@ -387,26 +383,26 @@
"print(\n",
" f\"The 'text' field value from the top hit is: '{hybrid_result['hits']['hits'][0]['_source']['text']}'\"\n",
")"
- ],
- "metadata": {
- "id": "bwR4jYvqqxTo"
- },
- "execution_count": null,
- "outputs": [],
- "id": "bwR4jYvqqxTo"
+ ]
},
{
"cell_type": "markdown",
- "source": [
- "## Test knn search using pre generated vector \n"
- ],
+ "id": "ltXYqp0qqz7R",
"metadata": {
"id": "ltXYqp0qqz7R"
},
- "id": "ltXYqp0qqz7R"
+ "source": [
+ "## Test knn search using pre generated vector \n"
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "id": "O5COtpTqq23t",
+ "metadata": {
+ "id": "O5COtpTqq23t"
+ },
+ "outputs": [],
"source": [
"# Generate embedding for tests\n",
"query_text = \"Hello\"\n",
@@ -428,26 +424,26 @@
"print(\n",
" f\"The 'text' field value from the top hit is: '{knn_result['hits']['hits'][0]['_source']['text']}'\"\n",
")"
- ],
- "metadata": {
- "id": "O5COtpTqq23t"
- },
- "execution_count": null,
- "outputs": [],
- "id": "O5COtpTqq23t"
+ ]
},
{
"cell_type": "markdown",
- "source": [
- "## Test source option"
- ],
+ "id": "0dnmimcJq42C",
"metadata": {
"id": "0dnmimcJq42C"
},
- "id": "0dnmimcJq42C"
+ "source": [
+ "## Test source option"
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "id": "v4_B72nHq7g1",
+ "metadata": {
+ "id": "v4_B72nHq7g1"
+ },
+ "outputs": [],
"source": [
"# Test `knn_search` method with model_id and query_text\n",
"query = \"Hello\"\n",
@@ -460,26 +456,26 @@
" query=query, model_id=model_id, k=2, source=False\n",
")\n",
"assert not \"_source\" in hybrid_result[\"hits\"][\"hits\"][0].keys()"
- ],
- "metadata": {
- "id": "v4_B72nHq7g1"
- },
- "execution_count": null,
- "outputs": [],
- "id": "v4_B72nHq7g1"
+ ]
},
{
"cell_type": "markdown",
- "source": [
- "## Test fields option "
- ],
+ "id": "teHgJgrlq-Jb",
"metadata": {
"id": "teHgJgrlq-Jb"
},
- "id": "teHgJgrlq-Jb"
+ "source": [
+ "## Test fields option "
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "id": "utNBbpZYrAYW",
+ "metadata": {
+ "id": "utNBbpZYrAYW"
+ },
+ "outputs": [],
"source": [
"# Test `knn_search` method with model_id and query_text\n",
"query = \"Hello\"\n",
@@ -492,72 +488,72 @@
" query=query, model_id=model_id, k=2, fields=[\"text\"]\n",
")\n",
"assert \"text\" in hybrid_result[\"hits\"][\"hits\"][0][\"fields\"].keys()"
- ],
- "metadata": {
- "id": "utNBbpZYrAYW"
- },
- "execution_count": null,
- "outputs": [],
- "id": "utNBbpZYrAYW"
+ ]
},
{
"cell_type": "markdown",
- "source": [
- "### Test with es client connection rather than cloud_id "
- ],
+ "id": "hddsIFferBy1",
"metadata": {
"id": "hddsIFferBy1"
},
- "id": "hddsIFferBy1"
+ "source": [
+ "### Test with es client connection rather than cloud_id "
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "id": "bXqrUnoirFia",
+ "metadata": {
+ "id": "bXqrUnoirFia"
+ },
+ "outputs": [],
"source": [
"# Create Elasticsearch connection\n",
"es_connection = Elasticsearch(\n",
" hosts=[\"https://es_cluster_url:port\"], basic_auth=(\"user\", \"password\")\n",
")"
- ],
- "metadata": {
- "id": "bXqrUnoirFia"
- },
- "execution_count": null,
- "outputs": [],
- "id": "bXqrUnoirFia"
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "id": "TIM__Hm8rSEW",
+ "metadata": {
+ "id": "TIM__Hm8rSEW"
+ },
+ "outputs": [],
"source": [
"# Instantiate ElasticsearchEmbeddings using es_connection\n",
"embeddings = ElasticsearchEmbeddings.from_es_connection(\n",
" model_id,\n",
" es_connection,\n",
")"
- ],
- "metadata": {
- "id": "TIM__Hm8rSEW"
- },
- "execution_count": null,
- "outputs": [],
- "id": "TIM__Hm8rSEW"
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "id": "1-CdnOrArVc_",
+ "metadata": {
+ "id": "1-CdnOrArVc_"
+ },
+ "outputs": [],
"source": [
"# Initialize ElasticKnnSearch\n",
"knn_search = ElasticKnnSearch(\n",
" es_connection=es_connection, index_name=test_index, embedding=embeddings\n",
")"
- ],
- "metadata": {
- "id": "1-CdnOrArVc_"
- },
- "execution_count": null,
- "outputs": [],
- "id": "1-CdnOrArVc_"
+ ]
},
{
"cell_type": "code",
+ "execution_count": null,
+ "id": "0kgyaL6QrYVF",
+ "metadata": {
+ "id": "0kgyaL6QrYVF"
+ },
+ "outputs": [],
"source": [
"# Test `knn_search` method with model_id and query_text\n",
"query = \"Hello\"\n",
@@ -566,16 +562,13 @@
"print(\n",
" f\"The 'text' field value from the top hit is: '{knn_result['hits']['hits'][0]['_source']['text']}'\"\n",
")"
- ],
- "metadata": {
- "id": "0kgyaL6QrYVF"
- },
- "execution_count": null,
- "outputs": [],
- "id": "0kgyaL6QrYVF"
+ ]
}
],
"metadata": {
+ "colab": {
+ "provenance": []
+ },
"kernelspec": {
"display_name": "Python 3 (ipykernel)",
"language": "python",
@@ -592,11 +585,8 @@
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
"version": "3.10.6"
- },
- "colab": {
- "provenance": []
}
},
"nbformat": 4,
"nbformat_minor": 5
-}
\ No newline at end of file
+}
diff --git a/docs/extras/modules/data_connection/vectorstores/integrations/hologres.ipynb b/docs/extras/modules/data_connection/vectorstores/integrations/hologres.ipynb
index 1d671cd6bded2..77ff7bf032e35 100644
--- a/docs/extras/modules/data_connection/vectorstores/integrations/hologres.ipynb
+++ b/docs/extras/modules/data_connection/vectorstores/integrations/hologres.ipynb
@@ -16,6 +16,15 @@
"Click [here](https://www.alibabacloud.com/zh/product/hologres) to fast deploy a Hologres cloud instance."
]
},
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#!pip install psycopg2"
+ ]
+ },
{
"cell_type": "code",
"execution_count": 1,
@@ -149,7 +158,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.9.16"
+ "version": "3.10.6"
}
},
"nbformat": 4,
diff --git a/docs/extras/modules/data_connection/vectorstores/integrations/mongodb_atlas_vector_search.ipynb b/docs/extras/modules/data_connection/vectorstores/integrations/mongodb_atlas.ipynb
similarity index 99%
rename from docs/extras/modules/data_connection/vectorstores/integrations/mongodb_atlas_vector_search.ipynb
rename to docs/extras/modules/data_connection/vectorstores/integrations/mongodb_atlas.ipynb
index ddb7f28fd9848..a56fc73cf50ec 100644
--- a/docs/extras/modules/data_connection/vectorstores/integrations/mongodb_atlas_vector_search.ipynb
+++ b/docs/extras/modules/data_connection/vectorstores/integrations/mongodb_atlas.ipynb
@@ -5,7 +5,7 @@
"id": "683953b3",
"metadata": {},
"source": [
- "# MongoDB Atlas Vector Search\n",
+ "# MongoDB Atlas\n",
"\n",
">[MongoDB Atlas](https://www.mongodb.com/docs/atlas/) is a fully-managed cloud database available in AWS , Azure, and GCP. It now has support for native Vector Search on your MongoDB document data.\n",
"\n",
@@ -214,7 +214,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.9.1"
+ "version": "3.10.6"
}
},
"nbformat": 4,
diff --git a/docs/extras/modules/data_connection/vectorstores/integrations/opensearch.ipynb b/docs/extras/modules/data_connection/vectorstores/integrations/opensearch.ipynb
index ee9fa2760e9e5..7d3d73136da0e 100644
--- a/docs/extras/modules/data_connection/vectorstores/integrations/opensearch.ipynb
+++ b/docs/extras/modules/data_connection/vectorstores/integrations/opensearch.ipynb
@@ -96,7 +96,7 @@
"id": "01a9a035",
"metadata": {},
"source": [
- "### similarity_search using Approximate k-NN\n",
+ "## similarity_search using Approximate k-NN\n",
"\n",
"`similarity_search` using `Approximate k-NN` Search with Custom Parameters"
]
@@ -182,7 +182,7 @@
"id": "0d0cd877",
"metadata": {},
"source": [
- "### similarity_search using Script Scoring\n",
+ "## similarity_search using Script Scoring\n",
"\n",
"`similarity_search` using `Script Scoring` with Custom Parameters"
]
@@ -221,7 +221,7 @@
"id": "a4af96cc",
"metadata": {},
"source": [
- "### similarity_search using Painless Scripting\n",
+ "## similarity_search using Painless Scripting\n",
"\n",
"`similarity_search` using `Painless Scripting` with Custom Parameters"
]
@@ -258,32 +258,35 @@
},
{
"cell_type": "markdown",
+ "id": "4f8fb0d0",
+ "metadata": {},
"source": [
- "### Maximum marginal relevance search (MMR)\n",
+ "## Maximum marginal relevance search (MMR)\n",
"If you’d like to look up for some similar documents, but you’d also like to receive diverse results, MMR is method you should consider. Maximal marginal relevance optimizes for similarity to query AND diversity among selected documents."
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "code",
"execution_count": null,
+ "id": "ba85e092",
+ "metadata": {
+ "collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
"docs = docsearch.max_marginal_relevance_search(query, k=2, fetch_k=10, lambda_param=0.5)"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "markdown",
"id": "73264864",
"metadata": {},
"source": [
- "### Using a preexisting OpenSearch instance\n",
+ "## Using a preexisting OpenSearch instance\n",
"\n",
"It's also possible to use a preexisting OpenSearch instance with documents that already have vectors present."
]
@@ -330,7 +333,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.11.3"
+ "version": "3.10.6"
}
},
"nbformat": 4,
diff --git a/docs/extras/modules/data_connection/vectorstores/integrations/pgvector.ipynb b/docs/extras/modules/data_connection/vectorstores/integrations/pgvector.ipynb
index 292ed6c813b11..381de0ee9f589 100644
--- a/docs/extras/modules/data_connection/vectorstores/integrations/pgvector.ipynb
+++ b/docs/extras/modules/data_connection/vectorstores/integrations/pgvector.ipynb
@@ -201,14 +201,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Similarity search with score"
- ]
- },
- {
- "cell_type": "markdown",
- "metadata": {},
- "source": [
- "### Similarity Search with Euclidean Distance (Default)"
+ "## Similarity Search with Euclidean Distance (Default)"
]
},
{
@@ -303,14 +296,14 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "## Working with vectorstore in PG"
+ "## Working with vectorstore"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Uploading a vectorstore in PG "
+ "### Uploading a vectorstore"
]
},
{
@@ -336,7 +329,7 @@
"cell_type": "markdown",
"metadata": {},
"source": [
- "### Retrieving a vectorstore in PG"
+ "### Retrieving a vectorstore"
]
},
{
@@ -498,7 +491,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.9.7"
+ "version": "3.10.6"
}
},
"nbformat": 4,
diff --git a/docs/extras/modules/data_connection/vectorstores/integrations/rockset_vector_database.ipynb b/docs/extras/modules/data_connection/vectorstores/integrations/rockset.ipynb
similarity index 91%
rename from docs/extras/modules/data_connection/vectorstores/integrations/rockset_vector_database.ipynb
rename to docs/extras/modules/data_connection/vectorstores/integrations/rockset.ipynb
index 0c44fa35797a6..bf96c786cd153 100644
--- a/docs/extras/modules/data_connection/vectorstores/integrations/rockset_vector_database.ipynb
+++ b/docs/extras/modules/data_connection/vectorstores/integrations/rockset.ipynb
@@ -1,20 +1,18 @@
{
"cells": [
{
- "attachments": {},
"cell_type": "markdown",
"id": "20b588b4",
"metadata": {},
"source": [
- "# Rockset Vector Search\n",
+ "# Rockset\n",
"\n",
- "[Rockset](https://rockset.com/product/) is a real-time analytics database service for serving low latency, high concurrency analytical queries at scale. It builds a Converged Index™ on structured and semi-structured data with an efficient store for vector embeddings. Its support for running SQL on schemaless data makes it a perfect choice for running vector search with metadata filters. \n",
+ ">[Rockset](https://rockset.com/product/) is a real-time analytics database service for serving low latency, high concurrency analytical queries at scale. It builds a Converged Index™ on structured and semi-structured data with an efficient store for vector embeddings. Its support for running SQL on schemaless data makes it a perfect choice for running vector search with metadata filters. \n",
"\n",
- "This notebook demonstrates how to use Rockset as a vectorstore in langchain. To get started, make sure you have a Rockset account and an API key available."
+ "This notebook demonstrates how to use `Rockset` as a vectorstore in langchain. To get started, make sure you have a `Rockset` account and an API key available."
]
},
{
- "attachments": {},
"cell_type": "markdown",
"id": "e290ddc0",
"metadata": {},
@@ -25,7 +23,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"id": "7d77bbbe",
"metadata": {},
@@ -52,7 +49,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"id": "7951c9cd",
"metadata": {},
@@ -71,7 +67,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"id": "8600900d",
"metadata": {},
@@ -80,12 +75,11 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"id": "3bf2f818",
"metadata": {},
"source": [
- "## Using Rockset langchain vectorstore"
+ "## Example"
]
},
{
@@ -109,7 +103,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"id": "474636a2",
"metadata": {},
@@ -138,7 +131,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"id": "1404cada",
"metadata": {},
@@ -173,7 +165,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"id": "f1290844",
"metadata": {},
@@ -205,7 +196,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"id": "5e15d630",
"metadata": {},
@@ -243,7 +233,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"id": "0765b822",
"metadata": {},
@@ -266,7 +255,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"id": "03fa12a9",
"metadata": {},
@@ -277,6 +265,14 @@
"\n",
"Keep an eye on https://rockset.com/blog/introducing-vector-search-on-rockset/ for future updates in this space!"
]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": null,
+ "id": "2763dddb-e87d-4d3b-b0bf-c246b0573d87",
+ "metadata": {},
+ "outputs": [],
+ "source": []
}
],
"metadata": {
@@ -295,7 +291,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.9.6"
+ "version": "3.10.6"
}
},
"nbformat": 4,
diff --git a/docs/extras/modules/data_connection/vectorstores/integrations/singlestoredb.ipynb b/docs/extras/modules/data_connection/vectorstores/integrations/singlestoredb.ipynb
index c011e95077839..a70370e82ee84 100644
--- a/docs/extras/modules/data_connection/vectorstores/integrations/singlestoredb.ipynb
+++ b/docs/extras/modules/data_connection/vectorstores/integrations/singlestoredb.ipynb
@@ -6,7 +6,9 @@
"metadata": {},
"source": [
"# SingleStoreDB\n",
- "[SingleStoreDB](https://singlestore.com/) is a high-performance distributed SQL database that supports deployment both in the [cloud](https://www.singlestore.com/cloud/) and on-premises. It provides vector storage, and vector functions including [dot_product](https://docs.singlestore.com/managed-service/en/reference/sql-reference/vector-functions/dot_product.html) and [euclidean_distance](https://docs.singlestore.com/managed-service/en/reference/sql-reference/vector-functions/euclidean_distance.html), thereby supporting AI applications that require text similarity matching. This tutorial illustrates how to [work with vector data in SingleStoreDB](https://docs.singlestore.com/managed-service/en/developer-resources/functional-extensions/working-with-vector-data.html)."
+ ">[SingleStoreDB](https://singlestore.com/) is a high-performance distributed SQL database that supports deployment both in the [cloud](https://www.singlestore.com/cloud/) and on-premises. It provides vector storage, and vector functions including [dot_product](https://docs.singlestore.com/managed-service/en/reference/sql-reference/vector-functions/dot_product.html) and [euclidean_distance](https://docs.singlestore.com/managed-service/en/reference/sql-reference/vector-functions/euclidean_distance.html), thereby supporting AI applications that require text similarity matching. \n",
+ "\n",
+ "This tutorial illustrates how to [work with vector data in SingleStoreDB](https://docs.singlestore.com/managed-service/en/developer-resources/functional-extensions/working-with-vector-data.html)."
]
},
{
@@ -129,7 +131,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.9.2"
+ "version": "3.10.6"
}
},
"nbformat": 4,
diff --git a/docs/extras/modules/data_connection/vectorstores/integrations/sklearn.ipynb b/docs/extras/modules/data_connection/vectorstores/integrations/sklearn.ipynb
index cca192ab47b00..b93c734a74f10 100644
--- a/docs/extras/modules/data_connection/vectorstores/integrations/sklearn.ipynb
+++ b/docs/extras/modules/data_connection/vectorstores/integrations/sklearn.ipynb
@@ -1,13 +1,12 @@
{
"cells": [
{
- "attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
- "# SKLearnVectorStore\n",
+ "# scikit-learn\n",
"\n",
- "[scikit-learn](https://scikit-learn.org/stable/) is an open source collection of machine learning algorithms, including some implementations of the [k nearest neighbors](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.NearestNeighbors.html). `SKLearnVectorStore` wraps this implementation and adds the possibility to persist the vector store in json, bson (binary json) or Apache Parquet format.\n",
+ ">[scikit-learn](https://scikit-learn.org/stable/) is an open source collection of machine learning algorithms, including some implementations of the [k nearest neighbors](https://scikit-learn.org/stable/modules/generated/sklearn.neighbors.NearestNeighbors.html). `SKLearnVectorStore` wraps this implementation and adds the possibility to persist the vector store in json, bson (binary json) or Apache Parquet format.\n",
"\n",
"This notebook shows how to use the `SKLearnVectorStore` vector database."
]
@@ -28,7 +27,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
@@ -48,7 +46,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
@@ -76,7 +73,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
@@ -120,7 +116,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
@@ -190,7 +185,6 @@
]
},
{
- "attachments": {},
"cell_type": "markdown",
"metadata": {},
"source": [
@@ -209,7 +203,7 @@
],
"metadata": {
"kernelspec": {
- "display_name": "sofia",
+ "display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
@@ -223,10 +217,9 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.8.16"
- },
- "orig_nbformat": 4
+ "version": "3.10.6"
+ }
},
"nbformat": 4,
- "nbformat_minor": 2
+ "nbformat_minor": 4
}
diff --git a/docs/extras/modules/data_connection/vectorstores/integrations/starrocks.ipynb b/docs/extras/modules/data_connection/vectorstores/integrations/starrocks.ipynb
index 84d640eb71dd2..515002a0bff24 100644
--- a/docs/extras/modules/data_connection/vectorstores/integrations/starrocks.ipynb
+++ b/docs/extras/modules/data_connection/vectorstores/integrations/starrocks.ipynb
@@ -7,11 +7,10 @@
"source": [
"# StarRocks\n",
"\n",
- "[StarRocks | A High-Performance Analytical Database](https://www.starrocks.io/)\n",
+ ">[StarRocks](https://www.starrocks.io/) is a High-Performance Analytical Database.\n",
+ "`StarRocks` is a next-gen sub-second MPP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics and ad-hoc query.\n",
"\n",
- "StarRocks is a next-gen sub-second MPP database for full analytics scenarios, including multi-dimensional analytics, real-time analytics and ad-hoc query.\n",
- "\n",
- "Usually StarRocks is categorized into OLAP, and it has showed excellent performance in [ClickBench — a Benchmark For Analytical DBMS](https://benchmark.clickhouse.com/). Since it has a super-fast vectorized execution engine, it could also be used as a fast vectordb.\n",
+ ">Usually `StarRocks` is categorized into OLAP, and it has showed excellent performance in [ClickBench — a Benchmark For Analytical DBMS](https://benchmark.clickhouse.com/). Since it has a super-fast vectorized execution engine, it could also be used as a fast vectordb.\n",
"\n",
"Here we'll show how to use the StarRocks Vector Store."
]
@@ -21,8 +20,17 @@
"id": "1685854f",
"metadata": {},
"source": [
- "\n",
- "## Import all used modules"
+ "## Setup"
+ ]
+ },
+ {
+ "cell_type": "code",
+ "execution_count": 1,
+ "id": "311d44bb-4aca-4f3b-8f97-5e1f29238e40",
+ "metadata": {},
+ "outputs": [],
+ "source": [
+ "#!pip install pymysql"
]
},
{
@@ -305,7 +313,7 @@
"name": "python",
"nbconvert_exporter": "python",
"pygments_lexer": "ipython3",
- "version": "3.11.3"
+ "version": "3.10.6"
}
},
"nbformat": 4,
diff --git a/docs/extras/modules/data_connection/vectorstores/integrations/tigris.ipynb b/docs/extras/modules/data_connection/vectorstores/integrations/tigris.ipynb
index e3718a669151a..ba529c1033b60 100644
--- a/docs/extras/modules/data_connection/vectorstores/integrations/tigris.ipynb
+++ b/docs/extras/modules/data_connection/vectorstores/integrations/tigris.ipynb
@@ -2,68 +2,67 @@
"cells": [
{
"cell_type": "markdown",
+ "metadata": {},
"source": [
"# Tigris\n",
"\n",
"> [Tigris](htttps://tigrisdata.com) is an open source Serverless NoSQL Database and Search Platform designed to simplify building high-performance vector search applications.\n",
- "> Tigris eliminates the infrastructure complexity of managing, operating, and synchronizing multiple tools, allowing you to focus on building great applications instead."
- ],
- "metadata": {
- "collapsed": false
- }
+ "> `Tigris` eliminates the infrastructure complexity of managing, operating, and synchronizing multiple tools, allowing you to focus on building great applications instead."
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {},
"source": [
"This notebook guides you how to use Tigris as your VectorStore"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {},
"source": [
"**Pre requisites**\n",
"1. An OpenAI account. You can sign up for an account [here](https://platform.openai.com/)\n",
"2. [Sign up for a free Tigris account](https://console.preview.tigrisdata.cloud). Once you have signed up for the Tigris account, create a new project called `vectordemo`. Next, make a note of the *Uri* for the region you've created your project in, the **clientId** and **clientSecret**. You can get all this information from the **Application Keys** section of the project."
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {},
"source": [
"Let's first install our dependencies:"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "code",
"execution_count": null,
+ "metadata": {
+ "collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
"!pip install tigrisdb openapi-schema-pydantic openai tiktoken"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {},
"source": [
"We will load the `OpenAI` api key and `Tigris` credentials in our environment"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "code",
"execution_count": null,
+ "metadata": {
+ "collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
"import os\n",
@@ -73,38 +72,42 @@
"os.environ[\"TIGRIS_PROJECT\"] = getpass.getpass(\"Tigris Project Name:\")\n",
"os.environ[\"TIGRIS_CLIENT_ID\"] = getpass.getpass(\"Tigris Client Id:\")\n",
"os.environ[\"TIGRIS_CLIENT_SECRET\"] = getpass.getpass(\"Tigris Client Secret:\")"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "code",
"execution_count": null,
+ "metadata": {
+ "collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
"from langchain.embeddings.openai import OpenAIEmbeddings\n",
"from langchain.text_splitter import CharacterTextSplitter\n",
"from langchain.vectorstores import Tigris\n",
"from langchain.document_loaders import TextLoader"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {},
"source": [
"### Initialize Tigris vector store\n",
"Let's import our test dataset:"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "code",
"execution_count": null,
+ "metadata": {
+ "collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
"loader = TextLoader(\"../../../state_of_the_union.txt\")\n",
@@ -113,87 +116,89 @@
"docs = text_splitter.split_documents(documents)\n",
"\n",
"embeddings = OpenAIEmbeddings()"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "code",
"execution_count": null,
+ "metadata": {
+ "collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
"vector_store = Tigris.from_documents(docs, embeddings, index_name=\"my_embeddings\")"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {},
"source": [
"### Similarity Search"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "code",
"execution_count": null,
+ "metadata": {
+ "collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
"found_docs = vector_store.similarity_search(query)\n",
"print(found_docs)"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {},
"source": [
"### Similarity Search with score (vector distance)"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "code",
"execution_count": null,
+ "metadata": {
+ "collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
"result = vector_store.similarity_search_with_score(query)\n",
"for doc, score in result:\n",
" print(f\"document={doc}, score={score}\")"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
}
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3",
+ "display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
- "version": 2
+ "version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
- "pygments_lexer": "ipython2",
- "version": "2.7.6"
+ "pygments_lexer": "ipython3",
+ "version": "3.10.6"
}
},
"nbformat": 4,
- "nbformat_minor": 0
+ "nbformat_minor": 4
}
diff --git a/docs/extras/modules/data_connection/vectorstores/integrations/typesense.ipynb b/docs/extras/modules/data_connection/vectorstores/integrations/typesense.ipynb
index a00fe58f73ab7..a547f5c640f0b 100644
--- a/docs/extras/modules/data_connection/vectorstores/integrations/typesense.ipynb
+++ b/docs/extras/modules/data_connection/vectorstores/integrations/typesense.ipynb
@@ -2,6 +2,7 @@
"cells": [
{
"cell_type": "markdown",
+ "metadata": {},
"source": [
"# Typesense\n",
"\n",
@@ -10,97 +11,105 @@
"> Typesense focuses on performance by storing the entire index in RAM (with a backup on disk) and also focuses on providing an out-of-the-box developer experience by simplifying available options and setting good defaults.\n",
">\n",
"> It also lets you combine attribute-based filtering together with vector queries, to fetch the most relevant documents."
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {},
"source": [
"This notebook shows you how to use Typesense as your VectorStore."
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {},
"source": [
"Let's first install our dependencies:"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "code",
"execution_count": null,
+ "metadata": {
+ "collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
"!pip install typesense openapi-schema-pydantic openai tiktoken"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {},
"source": [
"We want to use `OpenAIEmbeddings` so we have to get the OpenAI API Key."
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "code",
"execution_count": 2,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2023-05-23T22:48:02.968822Z",
+ "start_time": "2023-05-23T22:47:48.574094Z"
+ },
+ "collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
"import os\n",
"import getpass\n",
"\n",
"os.environ[\"OPENAI_API_KEY\"] = getpass.getpass(\"OpenAI API Key:\")"
- ],
- "metadata": {
- "collapsed": false,
- "ExecuteTime": {
- "end_time": "2023-05-23T22:48:02.968822Z",
- "start_time": "2023-05-23T22:47:48.574094Z"
- }
- }
+ ]
},
{
"cell_type": "code",
"execution_count": 6,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2023-05-23T22:50:34.775893Z",
+ "start_time": "2023-05-23T22:50:34.771889Z"
+ },
+ "collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
"from langchain.embeddings.openai import OpenAIEmbeddings\n",
"from langchain.text_splitter import CharacterTextSplitter\n",
"from langchain.vectorstores import Typesense\n",
"from langchain.document_loaders import TextLoader"
- ],
- "metadata": {
- "collapsed": false,
- "ExecuteTime": {
- "end_time": "2023-05-23T22:50:34.775893Z",
- "start_time": "2023-05-23T22:50:34.771889Z"
- }
- }
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {},
"source": [
"Let's import our test dataset:"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "code",
"execution_count": 19,
+ "metadata": {
+ "ExecuteTime": {
+ "end_time": "2023-05-23T22:56:19.093489Z",
+ "start_time": "2023-05-23T22:56:19.089Z"
+ },
+ "collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
"loader = TextLoader(\"../../../state_of_the_union.txt\")\n",
@@ -109,18 +118,17 @@
"docs = text_splitter.split_documents(documents)\n",
"\n",
"embeddings = OpenAIEmbeddings()"
- ],
- "metadata": {
- "collapsed": false,
- "ExecuteTime": {
- "end_time": "2023-05-23T22:56:19.093489Z",
- "start_time": "2023-05-23T22:56:19.089Z"
- }
- }
+ ]
},
{
"cell_type": "code",
"execution_count": null,
+ "metadata": {
+ "collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
"docsearch = Typesense.from_documents(\n",
@@ -134,98 +142,103 @@
" \"typesense_collection_name\": \"lang-chain\",\n",
" },\n",
")"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {},
"source": [
"## Similarity Search"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "code",
"execution_count": null,
+ "metadata": {
+ "collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
"found_docs = docsearch.similarity_search(query)"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "code",
"execution_count": null,
+ "metadata": {
+ "collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
"print(found_docs[0].page_content)"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "markdown",
+ "metadata": {},
"source": [
"## Typesense as a Retriever\n",
"\n",
"Typesense, as all the other vector stores, is a LangChain Retriever, by using cosine similarity."
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "code",
"execution_count": null,
+ "metadata": {
+ "collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
"retriever = docsearch.as_retriever()\n",
"retriever"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
},
{
"cell_type": "code",
"execution_count": null,
+ "metadata": {
+ "collapsed": false,
+ "jupyter": {
+ "outputs_hidden": false
+ }
+ },
"outputs": [],
"source": [
"query = \"What did the president say about Ketanji Brown Jackson\"\n",
"retriever.get_relevant_documents(query)[0]"
- ],
- "metadata": {
- "collapsed": false
- }
+ ]
}
],
"metadata": {
"kernelspec": {
- "display_name": "Python 3",
+ "display_name": "Python 3 (ipykernel)",
"language": "python",
"name": "python3"
},
"language_info": {
"codemirror_mode": {
"name": "ipython",
- "version": 2
+ "version": 3
},
"file_extension": ".py",
"mimetype": "text/x-python",
"name": "python",
"nbconvert_exporter": "python",
- "pygments_lexer": "ipython2",
- "version": "2.7.6"
+ "pygments_lexer": "ipython3",
+ "version": "3.10.6"
}
},
"nbformat": 4,
- "nbformat_minor": 0
+ "nbformat_minor": 4
}
diff --git a/langchain/agents/agent_toolkits/office365/__init__.py b/langchain/agents/agent_toolkits/office365/__init__.py
new file mode 100644
index 0000000000000..02e7f81659f5a
--- /dev/null
+++ b/langchain/agents/agent_toolkits/office365/__init__.py
@@ -0,0 +1 @@
+"""Gmail toolkit."""
diff --git a/langchain/agents/agent_toolkits/office365/toolkit.py b/langchain/agents/agent_toolkits/office365/toolkit.py
new file mode 100644
index 0000000000000..471a674bcfd4b
--- /dev/null
+++ b/langchain/agents/agent_toolkits/office365/toolkit.py
@@ -0,0 +1,38 @@
+from __future__ import annotations
+
+from typing import TYPE_CHECKING, List
+
+from pydantic import Field
+
+from langchain.agents.agent_toolkits.base import BaseToolkit
+from langchain.tools import BaseTool
+from langchain.tools.office365.create_draft_message import O365CreateDraftMessage
+from langchain.tools.office365.events_search import O365SearchEvents
+from langchain.tools.office365.messages_search import O365SearchEmails
+from langchain.tools.office365.send_event import O365SendEvent
+from langchain.tools.office365.send_message import O365SendMessage
+from langchain.tools.office365.utils import authenticate
+
+if TYPE_CHECKING:
+ from O365 import Account
+
+
+class O365Toolkit(BaseToolkit):
+ """Toolkit for interacting with Office365."""
+
+ account: Account = Field(default_factory=authenticate)
+
+ class Config:
+ """Pydantic config."""
+
+ arbitrary_types_allowed = True
+
+ def get_tools(self) -> List[BaseTool]:
+ """Get the tools in the toolkit."""
+ return [
+ O365SearchEvents(account=self.account),
+ O365CreateDraftMessage(account=self.account),
+ O365SearchEmails(account=self.account),
+ O365SendEvent(account=self.account),
+ O365SendMessage(account=self.account),
+ ]
diff --git a/langchain/agents/chat/output_parser.py b/langchain/agents/chat/output_parser.py
index 4da19526db84d..c7023426aa00e 100644
--- a/langchain/agents/chat/output_parser.py
+++ b/langchain/agents/chat/output_parser.py
@@ -17,13 +17,15 @@ def parse(self, text: str) -> Union[AgentAction, AgentFinish]:
try:
action = text.split("```")[1]
response = json.loads(action.strip())
- includes_action = "action" in response and "action_input" in response
+ includes_action = "action" in response
if includes_answer and includes_action:
raise OutputParserException(
"Parsing LLM output produced a final answer "
f"and a parse-able action: {text}"
)
- return AgentAction(response["action"], response["action_input"], text)
+ return AgentAction(
+ response["action"], response.get("action_input", {}), text
+ )
except Exception:
if not includes_answer:
diff --git a/langchain/agents/initialize.py b/langchain/agents/initialize.py
index cb26fb630a342..8b4ff608f70cc 100644
--- a/langchain/agents/initialize.py
+++ b/langchain/agents/initialize.py
@@ -51,7 +51,7 @@ def initialize_agent(
f"Got unknown agent type: {agent}. "
f"Valid types are: {AGENT_TO_CLASS.keys()}."
)
- tags_.append(agent.value)
+ tags_.append(agent.value if isinstance(agent, AgentType) else agent)
agent_cls = AGENT_TO_CLASS[agent]
agent_kwargs = agent_kwargs or {}
agent_obj = agent_cls.from_llm_and_tools(
diff --git a/langchain/agents/openai_functions_agent/base.py b/langchain/agents/openai_functions_agent/base.py
index bcfb234e09d4a..15c35b4de96a0 100644
--- a/langchain/agents/openai_functions_agent/base.py
+++ b/langchain/agents/openai_functions_agent/base.py
@@ -69,7 +69,7 @@ def _create_function_message(
"""
if not isinstance(observation, str):
try:
- content = json.dumps(observation)
+ content = json.dumps(observation, ensure_ascii=False)
except Exception:
content = str(observation)
else:
diff --git a/langchain/agents/openai_functions_multi_agent/base.py b/langchain/agents/openai_functions_multi_agent/base.py
index 81cd63db1bf08..4b4cbbbc6cbcc 100644
--- a/langchain/agents/openai_functions_multi_agent/base.py
+++ b/langchain/agents/openai_functions_multi_agent/base.py
@@ -68,7 +68,7 @@ def _create_function_message(
"""
if not isinstance(observation, str):
try:
- content = json.dumps(observation)
+ content = json.dumps(observation, ensure_ascii=False)
except Exception:
content = str(observation)
else:
diff --git a/langchain/cache.py b/langchain/cache.py
index 2cfb7ff37203e..db1718e6ffb11 100644
--- a/langchain/cache.py
+++ b/langchain/cache.py
@@ -226,7 +226,7 @@ def lookup(self, prompt: str, llm_string: str) -> Optional[RETURN_VAL_TYPE]:
def update(self, prompt: str, llm_string: str, return_val: RETURN_VAL_TYPE) -> None:
"""Update cache based on prompt and llm_string."""
for gen in return_val:
- if not isinstance(return_val, Generation):
+ if not isinstance(gen, Generation):
raise ValueError(
"RedisCache only supports caching of normal LLM generations, "
f"got {type(gen)}"
@@ -337,7 +337,7 @@ def lookup(self, prompt: str, llm_string: str) -> Optional[RETURN_VAL_TYPE]:
def update(self, prompt: str, llm_string: str, return_val: RETURN_VAL_TYPE) -> None:
"""Update cache based on prompt and llm_string."""
for gen in return_val:
- if not isinstance(return_val, Generation):
+ if not isinstance(gen, Generation):
raise ValueError(
"RedisSemanticCache only supports caching of "
f"normal LLM generations, got {type(gen)}"
@@ -455,7 +455,7 @@ def update(self, prompt: str, llm_string: str, return_val: RETURN_VAL_TYPE) -> N
and then store the `prompt` and `return_val` in the cache object.
"""
for gen in return_val:
- if not isinstance(return_val, Generation):
+ if not isinstance(gen, Generation):
raise ValueError(
"GPTCache only supports caching of normal LLM generations, "
f"got {type(gen)}"
@@ -628,7 +628,7 @@ def update(self, prompt: str, llm_string: str, return_val: RETURN_VAL_TYPE) -> N
Exception: Unexpected response
"""
for gen in return_val:
- if not isinstance(return_val, Generation):
+ if not isinstance(gen, Generation):
raise ValueError(
"Momento only supports caching of normal LLM generations, "
f"got {type(gen)}"
diff --git a/langchain/callbacks/arize_callback.py b/langchain/callbacks/arize_callback.py
index 7e1196e7c0d2f..62f952588a993 100644
--- a/langchain/callbacks/arize_callback.py
+++ b/langchain/callbacks/arize_callback.py
@@ -1,4 +1,3 @@
-import uuid
from datetime import datetime
from typing import Any, Dict, List, Optional, Union
@@ -33,6 +32,7 @@ def __init__(
self.prompt_tokens = 0
self.completion_tokens = 0
self.total_tokens = 0
+ self.step = 0
from arize.pandas.embeddings import EmbeddingGenerator, UseCases
from arize.pandas.logger import Client
@@ -84,11 +84,10 @@ def on_llm_end(self, response: LLMResult, **kwargs: Any) -> None:
self.total_tokens
) = self.completion_tokens = 0 # assign default value
- i = 0
-
for generations in response.generations:
for generation in generations:
- prompt = self.prompt_records[i]
+ prompt = self.prompt_records[self.step]
+ self.step = self.step + 1
prompt_embedding = pd.Series(
self.generator.generate_embeddings(
text_col=pd.Series(prompt.replace("\n", " "))
@@ -102,7 +101,6 @@ def on_llm_end(self, response: LLMResult, **kwargs: Any) -> None:
text_col=pd.Series(generation.text.replace("\n", " "))
).reset_index(drop=True)
)
- str(uuid.uuid4())
pred_timestamp = datetime.now().timestamp()
# Define the columns and data
@@ -165,8 +163,6 @@ def on_llm_end(self, response: LLMResult, **kwargs: Any) -> None:
else:
print(f'❌ Logging failed "{response_from_arize.text}"')
- i = i + 1
-
def on_llm_error(
self, error: Union[Exception, KeyboardInterrupt], **kwargs: Any
) -> None:
diff --git a/langchain/callbacks/manager.py b/langchain/callbacks/manager.py
index cbc58c4db802c..02bf28afccc94 100644
--- a/langchain/callbacks/manager.py
+++ b/langchain/callbacks/manager.py
@@ -74,7 +74,16 @@ def _get_debug() -> bool:
@contextmanager
def get_openai_callback() -> Generator[OpenAICallbackHandler, None, None]:
- """Get OpenAI callback handler in a context manager."""
+ """Get the OpenAI callback handler in a context manager.
+ which conveniently exposes token and cost information.
+
+ Returns:
+ OpenAICallbackHandler: The OpenAI callback handler.
+
+ Example:
+ >>> with get_openai_callback() as cb:
+ ... # Use the OpenAI callback handler
+ """
cb = OpenAICallbackHandler()
openai_callback_var.set(cb)
yield cb
@@ -85,7 +94,19 @@ def get_openai_callback() -> Generator[OpenAICallbackHandler, None, None]:
def tracing_enabled(
session_name: str = "default",
) -> Generator[TracerSessionV1, None, None]:
- """Get Tracer in a context manager."""
+ """Get the Deprecated LangChainTracer in a context manager.
+
+ Args:
+ session_name (str, optional): The name of the session.
+ Defaults to "default".
+
+ Returns:
+ TracerSessionV1: The LangChainTracer session.
+
+ Example:
+ >>> with tracing_enabled() as session:
+ ... # Use the LangChainTracer session
+ """
cb = LangChainTracerV1()
session = cast(TracerSessionV1, cb.load_session(session_name))
tracing_callback_var.set(cb)
@@ -97,7 +118,19 @@ def tracing_enabled(
def wandb_tracing_enabled(
session_name: str = "default",
) -> Generator[None, None, None]:
- """Get WandbTracer in a context manager."""
+ """Get the WandbTracer in a context manager.
+
+ Args:
+ session_name (str, optional): The name of the session.
+ Defaults to "default".
+
+ Returns:
+ None
+
+ Example:
+ >>> with wandb_tracing_enabled() as session:
+ ... # Use the WandbTracer session
+ """
cb = WandbTracer()
wandb_tracing_callback_var.set(cb)
yield None
@@ -110,7 +143,21 @@ def tracing_v2_enabled(
*,
example_id: Optional[Union[str, UUID]] = None,
) -> Generator[None, None, None]:
- """Get the experimental tracer handler in a context manager."""
+ """Instruct LangChain to log all runs in context to LangSmith.
+
+ Args:
+ project_name (str, optional): The name of the project.
+ Defaults to "default".
+ example_id (str or UUID, optional): The ID of the example.
+ Defaults to None.
+
+ Returns:
+ None
+
+ Example:
+ >>> with tracing_v2_enabled():
+ ... # LangChain code will automatically be traced
+ """
# Issue a warning that this is experimental
warnings.warn(
"The tracing v2 API is in development. "
@@ -133,14 +180,36 @@ def trace_as_chain_group(
*,
project_name: Optional[str] = None,
example_id: Optional[Union[str, UUID]] = None,
+ tags: Optional[List[str]] = None,
) -> Generator[CallbackManager, None, None]:
- """Get a callback manager for a chain group in a context manager."""
+ """Get a callback manager for a chain group in a context manager.
+ Useful for grouping different calls together as a single run even if
+ they aren't composed in a single chain.
+
+ Args:
+ group_name (str): The name of the chain group.
+ project_name (str, optional): The name of the project.
+ Defaults to None.
+ example_id (str or UUID, optional): The ID of the example.
+ Defaults to None.
+ tags (List[str], optional): The inheritable tags to apply to all runs.
+ Defaults to None.
+
+ Returns:
+ CallbackManager: The callback manager for the chain group.
+
+ Example:
+ >>> with trace_as_chain_group("group_name") as manager:
+ ... # Use the callback manager for the chain group
+ ... llm.predict("Foo", callbacks=manager)
+ """
cb = LangChainTracer(
project_name=project_name,
example_id=example_id,
)
cm = CallbackManager.configure(
inheritable_callbacks=[cb],
+ inheritable_tags=tags,
)
run_manager = cm.on_chain_start({"name": group_name}, {})
@@ -154,14 +223,34 @@ async def atrace_as_chain_group(
*,
project_name: Optional[str] = None,
example_id: Optional[Union[str, UUID]] = None,
+ tags: Optional[List[str]] = None,
) -> AsyncGenerator[AsyncCallbackManager, None]:
- """Get a callback manager for a chain group in a context manager."""
+ """Get an async callback manager for a chain group in a context manager.
+ Useful for grouping different async calls together as a single run even if
+ they aren't composed in a single chain.
+
+ Args:
+ group_name (str): The name of the chain group.
+ project_name (str, optional): The name of the project.
+ Defaults to None.
+ example_id (str or UUID, optional): The ID of the example.
+ Defaults to None.
+ tags (List[str], optional): The inheritable tags to apply to all runs.
+ Defaults to None.
+ Returns:
+ AsyncCallbackManager: The async callback manager for the chain group.
+
+ Example:
+ >>> async with atrace_as_chain_group("group_name") as manager:
+ ... # Use the async callback manager for the chain group
+ ... await llm.apredict("Foo", callbacks=manager)
+ """
cb = LangChainTracer(
project_name=project_name,
example_id=example_id,
)
cm = AsyncCallbackManager.configure(
- inheritable_callbacks=[cb],
+ inheritable_callbacks=[cb], inheritable_tags=tags
)
run_manager = await cm.on_chain_start({"name": group_name}, {})
@@ -293,7 +382,18 @@ def __init__(
tags: List[str],
inheritable_tags: List[str],
) -> None:
- """Initialize run manager."""
+ """Initialize the run manager.
+
+ Args:
+ run_id (UUID): The ID of the run.
+ handlers (List[BaseCallbackHandler]): The list of handlers.
+ inheritable_handlers (List[BaseCallbackHandler]):
+ The list of inheritable handlers.
+ parent_run_id (UUID, optional): The ID of the parent run.
+ Defaults to None.
+ tags (List[str]): The list of tags.
+ inheritable_tags (List[str]): The list of inheritable tags.
+ """
self.run_id = run_id
self.handlers = handlers
self.inheritable_handlers = inheritable_handlers
@@ -303,7 +403,11 @@ def __init__(
@classmethod
def get_noop_manager(cls: Type[BRM]) -> BRM:
- """Return a manager that doesn't perform any operations."""
+ """Return a manager that doesn't perform any operations.
+
+ Returns:
+ BaseRunManager: The noop manager.
+ """
return cls(
run_id=uuid4(),
handlers=[],
@@ -321,7 +425,14 @@ def on_text(
text: str,
**kwargs: Any,
) -> Any:
- """Run when text is received."""
+ """Run when text is received.
+
+ Args:
+ text (str): The received text.
+
+ Returns:
+ Any: The result of the callback.
+ """
_handle_event(
self.handlers,
"on_text",
@@ -341,7 +452,14 @@ async def on_text(
text: str,
**kwargs: Any,
) -> Any:
- """Run when text is received."""
+ """Run when text is received.
+
+ Args:
+ text (str): The received text.
+
+ Returns:
+ Any: The result of the callback.
+ """
await _ahandle_event(
self.handlers,
"on_text",
@@ -361,7 +479,11 @@ def on_llm_new_token(
token: str,
**kwargs: Any,
) -> None:
- """Run when LLM generates a new token."""
+ """Run when LLM generates a new token.
+
+ Args:
+ token (str): The new token.
+ """
_handle_event(
self.handlers,
"on_llm_new_token",
@@ -373,7 +495,11 @@ def on_llm_new_token(
)
def on_llm_end(self, response: LLMResult, **kwargs: Any) -> None:
- """Run when LLM ends running."""
+ """Run when LLM ends running.
+
+ Args:
+ response (LLMResult): The LLM result.
+ """
_handle_event(
self.handlers,
"on_llm_end",
@@ -389,7 +515,11 @@ def on_llm_error(
error: Union[Exception, KeyboardInterrupt],
**kwargs: Any,
) -> None:
- """Run when LLM errors."""
+ """Run when LLM errors.
+
+ Args:
+ error (Exception or KeyboardInterrupt): The error.
+ """
_handle_event(
self.handlers,
"on_llm_error",
@@ -409,7 +539,11 @@ async def on_llm_new_token(
token: str,
**kwargs: Any,
) -> None:
- """Run when LLM generates a new token."""
+ """Run when LLM generates a new token.
+
+ Args:
+ token (str): The new token.
+ """
await _ahandle_event(
self.handlers,
"on_llm_new_token",
@@ -421,7 +555,11 @@ async def on_llm_new_token(
)
async def on_llm_end(self, response: LLMResult, **kwargs: Any) -> None:
- """Run when LLM ends running."""
+ """Run when LLM ends running.
+
+ Args:
+ response (LLMResult): The LLM result.
+ """
await _ahandle_event(
self.handlers,
"on_llm_end",
@@ -437,7 +575,11 @@ async def on_llm_error(
error: Union[Exception, KeyboardInterrupt],
**kwargs: Any,
) -> None:
- """Run when LLM errors."""
+ """Run when LLM errors.
+
+ Args:
+ error (Exception or KeyboardInterrupt): The error.
+ """
await _ahandle_event(
self.handlers,
"on_llm_error",
@@ -453,7 +595,15 @@ class CallbackManagerForChainRun(RunManager, ChainManagerMixin):
"""Callback manager for chain run."""
def get_child(self, tag: Optional[str] = None) -> CallbackManager:
- """Get a child callback manager."""
+ """Get a child callback manager.
+
+ Args:
+ tag (str, optional): The tag for the child callback manager.
+ Defaults to None.
+
+ Returns:
+ CallbackManager: The child callback manager.
+ """
manager = CallbackManager(handlers=[], parent_run_id=self.run_id)
manager.set_handlers(self.inheritable_handlers)
manager.add_tags(self.inheritable_tags)
@@ -462,7 +612,11 @@ def get_child(self, tag: Optional[str] = None) -> CallbackManager:
return manager
def on_chain_end(self, outputs: Dict[str, Any], **kwargs: Any) -> None:
- """Run when chain ends running."""
+ """Run when chain ends running.
+
+ Args:
+ outputs (Dict[str, Any]): The outputs of the chain.
+ """
_handle_event(
self.handlers,
"on_chain_end",
@@ -478,7 +632,11 @@ def on_chain_error(
error: Union[Exception, KeyboardInterrupt],
**kwargs: Any,
) -> None:
- """Run when chain errors."""
+ """Run when chain errors.
+
+ Args:
+ error (Exception or KeyboardInterrupt): The error.
+ """
_handle_event(
self.handlers,
"on_chain_error",
@@ -490,7 +648,14 @@ def on_chain_error(
)
def on_agent_action(self, action: AgentAction, **kwargs: Any) -> Any:
- """Run when agent action is received."""
+ """Run when agent action is received.
+
+ Args:
+ action (AgentAction): The agent action.
+
+ Returns:
+ Any: The result of the callback.
+ """
_handle_event(
self.handlers,
"on_agent_action",
@@ -502,7 +667,14 @@ def on_agent_action(self, action: AgentAction, **kwargs: Any) -> Any:
)
def on_agent_finish(self, finish: AgentFinish, **kwargs: Any) -> Any:
- """Run when agent finish is received."""
+ """Run when agent finish is received.
+
+ Args:
+ finish (AgentFinish): The agent finish.
+
+ Returns:
+ Any: The result of the callback.
+ """
_handle_event(
self.handlers,
"on_agent_finish",
@@ -518,7 +690,15 @@ class AsyncCallbackManagerForChainRun(AsyncRunManager, ChainManagerMixin):
"""Async callback manager for chain run."""
def get_child(self, tag: Optional[str] = None) -> AsyncCallbackManager:
- """Get a child callback manager."""
+ """Get a child callback manager.
+
+ Args:
+ tag (str, optional): The tag for the child callback manager.
+ Defaults to None.
+
+ Returns:
+ AsyncCallbackManager: The child callback manager.
+ """
manager = AsyncCallbackManager(handlers=[], parent_run_id=self.run_id)
manager.set_handlers(self.inheritable_handlers)
manager.add_tags(self.inheritable_tags)
@@ -527,7 +707,11 @@ def get_child(self, tag: Optional[str] = None) -> AsyncCallbackManager:
return manager
async def on_chain_end(self, outputs: Dict[str, Any], **kwargs: Any) -> None:
- """Run when chain ends running."""
+ """Run when chain ends running.
+
+ Args:
+ outputs (Dict[str, Any]): The outputs of the chain.
+ """
await _ahandle_event(
self.handlers,
"on_chain_end",
@@ -543,7 +727,11 @@ async def on_chain_error(
error: Union[Exception, KeyboardInterrupt],
**kwargs: Any,
) -> None:
- """Run when chain errors."""
+ """Run when chain errors.
+
+ Args:
+ error (Exception or KeyboardInterrupt): The error.
+ """
await _ahandle_event(
self.handlers,
"on_chain_error",
@@ -555,7 +743,14 @@ async def on_chain_error(
)
async def on_agent_action(self, action: AgentAction, **kwargs: Any) -> Any:
- """Run when agent action is received."""
+ """Run when agent action is received.
+
+ Args:
+ action (AgentAction): The agent action.
+
+ Returns:
+ Any: The result of the callback.
+ """
await _ahandle_event(
self.handlers,
"on_agent_action",
@@ -567,7 +762,14 @@ async def on_agent_action(self, action: AgentAction, **kwargs: Any) -> Any:
)
async def on_agent_finish(self, finish: AgentFinish, **kwargs: Any) -> Any:
- """Run when agent finish is received."""
+ """Run when agent finish is received.
+
+ Args:
+ finish (AgentFinish): The agent finish.
+
+ Returns:
+ Any: The result of the callback.
+ """
await _ahandle_event(
self.handlers,
"on_agent_finish",
@@ -583,7 +785,15 @@ class CallbackManagerForToolRun(RunManager, ToolManagerMixin):
"""Callback manager for tool run."""
def get_child(self, tag: Optional[str] = None) -> CallbackManager:
- """Get a child callback manager."""
+ """Get a child callback manager.
+
+ Args:
+ tag (str, optional): The tag for the child callback manager.
+ Defaults to None.
+
+ Returns:
+ CallbackManager: The child callback manager.
+ """
manager = CallbackManager(handlers=[], parent_run_id=self.run_id)
manager.set_handlers(self.inheritable_handlers)
manager.add_tags(self.inheritable_tags)
@@ -596,7 +806,11 @@ def on_tool_end(
output: str,
**kwargs: Any,
) -> None:
- """Run when tool ends running."""
+ """Run when tool ends running.
+
+ Args:
+ output (str): The output of the tool.
+ """
_handle_event(
self.handlers,
"on_tool_end",
@@ -612,7 +826,11 @@ def on_tool_error(
error: Union[Exception, KeyboardInterrupt],
**kwargs: Any,
) -> None:
- """Run when tool errors."""
+ """Run when tool errors.
+
+ Args:
+ error (Exception or KeyboardInterrupt): The error.
+ """
_handle_event(
self.handlers,
"on_tool_error",
@@ -628,7 +846,15 @@ class AsyncCallbackManagerForToolRun(AsyncRunManager, ToolManagerMixin):
"""Async callback manager for tool run."""
def get_child(self, tag: Optional[str] = None) -> AsyncCallbackManager:
- """Get a child callback manager."""
+ """Get a child callback manager.
+
+ Args:
+ tag (str, optional): The tag to add to the child
+ callback manager. Defaults to None.
+
+ Returns:
+ AsyncCallbackManager: The child callback manager.
+ """
manager = AsyncCallbackManager(handlers=[], parent_run_id=self.run_id)
manager.set_handlers(self.inheritable_handlers)
manager.add_tags(self.inheritable_tags)
@@ -637,7 +863,11 @@ def get_child(self, tag: Optional[str] = None) -> AsyncCallbackManager:
return manager
async def on_tool_end(self, output: str, **kwargs: Any) -> None:
- """Run when tool ends running."""
+ """Run when tool ends running.
+
+ Args:
+ output (str): The output of the tool.
+ """
await _ahandle_event(
self.handlers,
"on_tool_end",
@@ -653,7 +883,11 @@ async def on_tool_error(
error: Union[Exception, KeyboardInterrupt],
**kwargs: Any,
) -> None:
- """Run when tool errors."""
+ """Run when tool errors.
+
+ Args:
+ error (Exception or KeyboardInterrupt): The error.
+ """
await _ahandle_event(
self.handlers,
"on_tool_error",
@@ -674,7 +908,17 @@ def on_llm_start(
prompts: List[str],
**kwargs: Any,
) -> List[CallbackManagerForLLMRun]:
- """Run when LLM starts running."""
+ """Run when LLM starts running.
+
+ Args:
+ serialized (Dict[str, Any]): The serialized LLM.
+ prompts (List[str]): The list of prompts.
+ run_id (UUID, optional): The ID of the run. Defaults to None.
+
+ Returns:
+ List[CallbackManagerForLLMRun]: A callback manager for each
+ prompt as an LLM run.
+ """
managers = []
for prompt in prompts:
run_id_ = uuid4()
@@ -709,7 +953,17 @@ def on_chat_model_start(
messages: List[List[BaseMessage]],
**kwargs: Any,
) -> List[CallbackManagerForLLMRun]:
- """Run when LLM starts running."""
+ """Run when LLM starts running.
+
+ Args:
+ serialized (Dict[str, Any]): The serialized LLM.
+ messages (List[List[BaseMessage]]): The list of messages.
+ run_id (UUID, optional): The ID of the run. Defaults to None.
+
+ Returns:
+ List[CallbackManagerForLLMRun]: A callback manager for each
+ list of messages as an LLM run.
+ """
managers = []
for message_list in messages:
@@ -746,7 +1000,16 @@ def on_chain_start(
run_id: Optional[UUID] = None,
**kwargs: Any,
) -> CallbackManagerForChainRun:
- """Run when chain starts running."""
+ """Run when chain starts running.
+
+ Args:
+ serialized (Dict[str, Any]): The serialized chain.
+ inputs (Dict[str, Any]): The inputs to the chain.
+ run_id (UUID, optional): The ID of the run. Defaults to None.
+
+ Returns:
+ CallbackManagerForChainRun: The callback manager for the chain run.
+ """
if run_id is None:
run_id = uuid4()
@@ -779,7 +1042,17 @@ def on_tool_start(
parent_run_id: Optional[UUID] = None,
**kwargs: Any,
) -> CallbackManagerForToolRun:
- """Run when tool starts running."""
+ """Run when tool starts running.
+
+ Args:
+ serialized (Dict[str, Any]): The serialized tool.
+ input_str (str): The input to the tool.
+ run_id (UUID, optional): The ID of the run. Defaults to None.
+ parent_run_id (UUID, optional): The ID of the parent run. Defaults to None.
+
+ Returns:
+ CallbackManagerForToolRun: The callback manager for the tool run.
+ """
if run_id is None:
run_id = uuid4()
@@ -813,7 +1086,22 @@ def configure(
inheritable_tags: Optional[List[str]] = None,
local_tags: Optional[List[str]] = None,
) -> CallbackManager:
- """Configure the callback manager."""
+ """Configure the callback manager.
+
+ Args:
+ inheritable_callbacks (Optional[Callbacks], optional): The inheritable
+ callbacks. Defaults to None.
+ local_callbacks (Optional[Callbacks], optional): The local callbacks.
+ Defaults to None.
+ verbose (bool, optional): Whether to enable verbose mode. Defaults to False.
+ inheritable_tags (Optional[List[str]], optional): The inheritable tags.
+ Defaults to None.
+ local_tags (Optional[List[str]], optional): The local tags.
+ Defaults to None.
+
+ Returns:
+ CallbackManager: The configured callback manager.
+ """
return _configure(
cls,
inheritable_callbacks,
@@ -838,7 +1126,18 @@ async def on_llm_start(
prompts: List[str],
**kwargs: Any,
) -> List[AsyncCallbackManagerForLLMRun]:
- """Run when LLM starts running."""
+ """Run when LLM starts running.
+
+ Args:
+ serialized (Dict[str, Any]): The serialized LLM.
+ prompts (List[str]): The list of prompts.
+ run_id (UUID, optional): The ID of the run. Defaults to None.
+
+ Returns:
+ List[AsyncCallbackManagerForLLMRun]: The list of async
+ callback managers, one for each LLM Run corresponding
+ to each prompt.
+ """
tasks = []
managers = []
@@ -881,6 +1180,18 @@ async def on_chat_model_start(
messages: List[List[BaseMessage]],
**kwargs: Any,
) -> Any:
+ """Run when LLM starts running.
+
+ Args:
+ serialized (Dict[str, Any]): The serialized LLM.
+ messages (List[List[BaseMessage]]): The list of messages.
+ run_id (UUID, optional): The ID of the run. Defaults to None.
+
+ Returns:
+ List[AsyncCallbackManagerForLLMRun]: The list of
+ async callback managers, one for each LLM Run
+ corresponding to each inner message list.
+ """
tasks = []
managers = []
@@ -922,7 +1233,17 @@ async def on_chain_start(
run_id: Optional[UUID] = None,
**kwargs: Any,
) -> AsyncCallbackManagerForChainRun:
- """Run when chain starts running."""
+ """Run when chain starts running.
+
+ Args:
+ serialized (Dict[str, Any]): The serialized chain.
+ inputs (Dict[str, Any]): The inputs to the chain.
+ run_id (UUID, optional): The ID of the run. Defaults to None.
+
+ Returns:
+ AsyncCallbackManagerForChainRun: The async callback manager
+ for the chain run.
+ """
if run_id is None:
run_id = uuid4()
@@ -955,7 +1276,19 @@ async def on_tool_start(
parent_run_id: Optional[UUID] = None,
**kwargs: Any,
) -> AsyncCallbackManagerForToolRun:
- """Run when tool starts running."""
+ """Run when tool starts running.
+
+ Args:
+ serialized (Dict[str, Any]): The serialized tool.
+ input_str (str): The input to the tool.
+ run_id (UUID, optional): The ID of the run. Defaults to None.
+ parent_run_id (UUID, optional): The ID of the parent run.
+ Defaults to None.
+
+ Returns:
+ AsyncCallbackManagerForToolRun: The async callback manager
+ for the tool run.
+ """
if run_id is None:
run_id = uuid4()
@@ -989,7 +1322,22 @@ def configure(
inheritable_tags: Optional[List[str]] = None,
local_tags: Optional[List[str]] = None,
) -> AsyncCallbackManager:
- """Configure the callback manager."""
+ """Configure the async callback manager.
+
+ Args:
+ inheritable_callbacks (Optional[Callbacks], optional): The inheritable
+ callbacks. Defaults to None.
+ local_callbacks (Optional[Callbacks], optional): The local callbacks.
+ Defaults to None.
+ verbose (bool, optional): Whether to enable verbose mode. Defaults to False.
+ inheritable_tags (Optional[List[str]], optional): The inheritable tags.
+ Defaults to None.
+ local_tags (Optional[List[str]], optional): The local tags.
+ Defaults to None.
+
+ Returns:
+ AsyncCallbackManager: The configured async callback manager.
+ """
return _configure(
cls,
inheritable_callbacks,
@@ -1004,7 +1352,14 @@ def configure(
def env_var_is_set(env_var: str) -> bool:
- """Check if an environment variable is set."""
+ """Check if an environment variable is set.
+
+ Args:
+ env_var (str): The name of the environment variable.
+
+ Returns:
+ bool: True if the environment variable is set, False otherwise.
+ """
return env_var in os.environ and os.environ[env_var] not in (
"",
"0",
@@ -1021,7 +1376,22 @@ def _configure(
inheritable_tags: Optional[List[str]] = None,
local_tags: Optional[List[str]] = None,
) -> T:
- """Configure the callback manager."""
+ """Configure the callback manager.
+
+ Args:
+ callback_manager_cls (Type[T]): The callback manager class.
+ inheritable_callbacks (Optional[Callbacks], optional): The inheritable
+ callbacks. Defaults to None.
+ local_callbacks (Optional[Callbacks], optional): The local callbacks.
+ Defaults to None.
+ verbose (bool, optional): Whether to enable verbose mode. Defaults to False.
+ inheritable_tags (Optional[List[str]], optional): The inheritable tags.
+ Defaults to None.
+ local_tags (Optional[List[str]], optional): The local tags. Defaults to None.
+
+ Returns:
+ T: The configured callback manager.
+ """
callback_manager = callback_manager_cls(handlers=[])
if inheritable_callbacks or local_callbacks:
if isinstance(inheritable_callbacks, list) or inheritable_callbacks is None:
diff --git a/langchain/callbacks/mlflow_callback.py b/langchain/callbacks/mlflow_callback.py
index 34b05e0e3a794..8bae7739f41a0 100644
--- a/langchain/callbacks/mlflow_callback.py
+++ b/langchain/callbacks/mlflow_callback.py
@@ -118,7 +118,7 @@ class MlflowLogger:
Parameters:
name (str): Name of the run.
experiment (str): Name of the experiment.
- tags (str): Tags to be attached for the run.
+ tags (dict): Tags to be attached for the run.
tracking_uri (str): MLflow tracking server uri.
This handler implements the helper functions to initialize,
@@ -223,7 +223,7 @@ class MlflowCallbackHandler(BaseMetadataCallbackHandler, BaseCallbackHandler):
Parameters:
name (str): Name of the run.
experiment (str): Name of the experiment.
- tags (str): Tags to be attached for the run.
+ tags (dict): Tags to be attached for the run.
tracking_uri (str): MLflow tracking server uri.
This handler will utilize the associated callback method called and formats
diff --git a/langchain/callbacks/openai_info.py b/langchain/callbacks/openai_info.py
index c66ec7ce8730a..549720e7c4d82 100644
--- a/langchain/callbacks/openai_info.py
+++ b/langchain/callbacks/openai_info.py
@@ -32,6 +32,7 @@
"gpt-3.5-turbo-16k-completion": 0.004,
"gpt-3.5-turbo-16k-0613-completion": 0.004,
# Others
+ "gpt-35-turbo": 0.002, # Azure OpenAI version of ChatGPT
"text-ada-001": 0.0004,
"ada": 0.0004,
"text-babbage-001": 0.0005,
diff --git a/langchain/callbacks/tracers/base.py b/langchain/callbacks/tracers/base.py
index dd0c1183558d0..6cfad4d431e9d 100644
--- a/langchain/callbacks/tracers/base.py
+++ b/langchain/callbacks/tracers/base.py
@@ -1,6 +1,7 @@
"""Base interfaces for tracing runs."""
from __future__ import annotations
+import logging
from abc import ABC, abstractmethod
from datetime import datetime
from typing import Any, Dict, List, Optional, Union
@@ -10,6 +11,8 @@
from langchain.callbacks.tracers.schemas import Run, RunTypeEnum
from langchain.schema import LLMResult
+logger = logging.getLogger(__name__)
+
class TracerException(Exception):
"""Base class for exceptions in tracers module."""
@@ -41,9 +44,7 @@ def _start_trace(self, run: Run) -> None:
if parent_run:
self._add_child_run(parent_run, run)
else:
- raise TracerException(
- f"Parent run with UUID {run.parent_run_id} not found."
- )
+ logger.warning(f"Parent run with UUID {run.parent_run_id} not found.")
self.run_map[str(run.id)] = run
def _end_trace(self, run: Run) -> None:
@@ -53,10 +54,8 @@ def _end_trace(self, run: Run) -> None:
else:
parent_run = self.run_map.get(str(run.parent_run_id))
if parent_run is None:
- raise TracerException(
- f"Parent run with UUID {run.parent_run_id} not found."
- )
- if (
+ logger.warning(f"Parent run with UUID {run.parent_run_id} not found.")
+ elif (
run.child_execution_order is not None
and parent_run.child_execution_order is not None
and run.child_execution_order > parent_run.child_execution_order
@@ -71,7 +70,8 @@ def _get_execution_order(self, parent_run_id: Optional[str] = None) -> int:
parent_run = self.run_map.get(parent_run_id)
if parent_run is None:
- raise TracerException(f"Parent run with UUID {parent_run_id} not found.")
+ logger.warning(f"Parent run with UUID {parent_run_id} not found.")
+ return 1
if parent_run.child_execution_order is None:
raise TracerException(
f"Parent run with UUID {parent_run_id} has no child execution order."
diff --git a/langchain/chains/openai_functions/openapi.py b/langchain/chains/openai_functions/openapi.py
index 3d58c052aca1c..5a29bb687d837 100644
--- a/langchain/chains/openai_functions/openapi.py
+++ b/langchain/chains/openai_functions/openapi.py
@@ -161,7 +161,7 @@ def default_call_api(name: str, fn_args: dict, **kwargs: Any) -> Any:
method = _name_to_call_map[name]["method"]
url = _name_to_call_map[name]["url"]
path_params = fn_args.pop("path_params", {})
- _format_url(url, path_params)
+ url = _format_url(url, path_params)
if "data" in fn_args and isinstance(fn_args["data"], dict):
fn_args["data"] = json.dumps(fn_args["data"])
_kwargs = {**fn_args, **kwargs}
diff --git a/langchain/chains/query_constructor/ir.py b/langchain/chains/query_constructor/ir.py
index 99d26ce0f7f9f..c7f6581f72a1a 100644
--- a/langchain/chains/query_constructor/ir.py
+++ b/langchain/chains/query_constructor/ir.py
@@ -75,8 +75,8 @@ class Comparator(str, Enum):
GTE = "gte"
LT = "lt"
LTE = "lte"
- CONTAIN = "contain"
- LIKE = "like"
+ CONTAIN = "list_contain"
+ LIKE = "string_pattern_like"
class FilterDirective(Expr, ABC):
diff --git a/langchain/chat_models/openai.py b/langchain/chat_models/openai.py
index 83e43ae0ff11b..f1725b83313c3 100644
--- a/langchain/chat_models/openai.py
+++ b/langchain/chat_models/openai.py
@@ -184,6 +184,16 @@ def lc_serializable(self) -> bool:
"""Number of chat completions to generate for each prompt."""
max_tokens: Optional[int] = None
"""Maximum number of tokens to generate."""
+ tiktoken_model_name: Optional[str] = None
+ """The model name to pass to tiktoken when using this class.
+ Tiktoken is used to count the number of tokens in documents to constrain
+ them to be under a certain limit. By default, when set to None, this will
+ be the same as the embedding model name. However, there are some cases
+ where you may want to use this Embedding class with a model name not
+ supported by tiktoken. This can include when using Azure embeddings or
+ when using one of the many model providers that expose an OpenAI-like
+ API but with different models. In those cases, in order to avoid erroring
+ when tiktoken is called, you can specify a model name to use here."""
class Config:
"""Configuration for this pydantic object."""
@@ -448,15 +458,18 @@ def _llm_type(self) -> str:
def _get_encoding_model(self) -> Tuple[str, tiktoken.Encoding]:
tiktoken_ = _import_tiktoken()
- model = self.model_name
- if model == "gpt-3.5-turbo":
- # gpt-3.5-turbo may change over time.
- # Returning num tokens assuming gpt-3.5-turbo-0301.
- model = "gpt-3.5-turbo-0301"
- elif model == "gpt-4":
- # gpt-4 may change over time.
- # Returning num tokens assuming gpt-4-0314.
- model = "gpt-4-0314"
+ if self.tiktoken_model_name is not None:
+ model = self.tiktoken_model_name
+ else:
+ model = self.model_name
+ if model == "gpt-3.5-turbo":
+ # gpt-3.5-turbo may change over time.
+ # Returning num tokens assuming gpt-3.5-turbo-0301.
+ model = "gpt-3.5-turbo-0301"
+ elif model == "gpt-4":
+ # gpt-4 may change over time.
+ # Returning num tokens assuming gpt-4-0314.
+ model = "gpt-4-0314"
# Returns the number of tokens used by a list of messages.
try:
encoding = tiktoken_.encoding_for_model(model)
diff --git a/langchain/document_loaders/__init__.py b/langchain/document_loaders/__init__.py
index 57e450c884429..2bf8f76e76441 100644
--- a/langchain/document_loaders/__init__.py
+++ b/langchain/document_loaders/__init__.py
@@ -95,7 +95,7 @@
from langchain.document_loaders.pyspark_dataframe import PySparkDataFrameLoader
from langchain.document_loaders.python import PythonLoader
from langchain.document_loaders.readthedocs import ReadTheDocsLoader
-from langchain.document_loaders.recursive_url_loader import RecusiveUrlLoader
+from langchain.document_loaders.recursive_url_loader import RecursiveUrlLoader
from langchain.document_loaders.reddit import RedditPostsLoader
from langchain.document_loaders.roam import RoamLoader
from langchain.document_loaders.rst import UnstructuredRSTLoader
@@ -230,7 +230,7 @@
"PySparkDataFrameLoader",
"PythonLoader",
"ReadTheDocsLoader",
- "RecusiveUrlLoader",
+ "RecursiveUrlLoader",
"RedditPostsLoader",
"RoamLoader",
"S3DirectoryLoader",
diff --git a/langchain/document_loaders/notiondb.py b/langchain/document_loaders/notiondb.py
index f43fd5f496548..9a666eeab5528 100644
--- a/langchain/document_loaders/notiondb.py
+++ b/langchain/document_loaders/notiondb.py
@@ -48,13 +48,13 @@ def load(self) -> List[Document]:
Returns:
List[Document]: List of documents.
"""
- page_ids = self._retrieve_page_ids()
+ page_summaries = self._retrieve_page_summaries()
- return list(self.load_page(page_id) for page_id in page_ids)
+ return list(self.load_page(page_summary) for page_summary in page_summaries)
- def _retrieve_page_ids(
+ def _retrieve_page_summaries(
self, query_dict: Dict[str, Any] = {"page_size": 100}
- ) -> List[str]:
+ ) -> List[Dict[str, Any]]:
"""Get all the pages from a Notion database."""
pages: List[Dict[str, Any]] = []
@@ -72,18 +72,16 @@ def _retrieve_page_ids(
query_dict["start_cursor"] = data.get("next_cursor")
- page_ids = [page["id"] for page in pages]
+ return pages
- return page_ids
-
- def load_page(self, page_id: str) -> Document:
+ def load_page(self, page_summary: Dict[str, Any]) -> Document:
"""Read a page."""
- data = self._request(PAGE_URL.format(page_id=page_id))
+ page_id = page_summary["id"]
# load properties as metadata
metadata: Dict[str, Any] = {}
- for prop_name, prop_data in data["properties"].items():
+ for prop_name, prop_data in page_summary["properties"].items():
prop_type = prop_data["type"]
if prop_type == "rich_text":
diff --git a/langchain/document_loaders/recursive_url_loader.py b/langchain/document_loaders/recursive_url_loader.py
index 7107f3a7345ea..b1a0250d74fcd 100644
--- a/langchain/document_loaders/recursive_url_loader.py
+++ b/langchain/document_loaders/recursive_url_loader.py
@@ -7,7 +7,7 @@
from langchain.document_loaders.base import BaseLoader
-class RecusiveUrlLoader(BaseLoader):
+class RecursiveUrlLoader(BaseLoader):
"""Loader that loads all child links from a given url."""
def __init__(self, url: str, exclude_dirs: Optional[str] = None) -> None:
@@ -24,7 +24,7 @@ def get_child_links_recursive(
from bs4 import BeautifulSoup
except ImportError:
raise ImportError(
- "The BeautifulSoup package is required for the RecusiveUrlLoader."
+ "The BeautifulSoup package is required for the RecursiveUrlLoader."
)
# Construct the base and parent URLs
diff --git a/langchain/embeddings/openai.py b/langchain/embeddings/openai.py
index 9c23323075242..f3cb66547b7ed 100644
--- a/langchain/embeddings/openai.py
+++ b/langchain/embeddings/openai.py
@@ -170,6 +170,16 @@ class OpenAIEmbeddings(BaseModel, Embeddings):
request_timeout: Optional[Union[float, Tuple[float, float]]] = None
"""Timeout in seconds for the OpenAPI request."""
headers: Any = None
+ tiktoken_model_name: Optional[str] = None
+ """The model name to pass to tiktoken when using this class.
+ Tiktoken is used to count the number of tokens in documents to constrain
+ them to be under a certain limit. By default, when set to None, this will
+ be the same as the embedding model name. However, there are some cases
+ where you may want to use this Embedding class with a model name not
+ supported by tiktoken. This can include when using Azure embeddings or
+ when using one of the many model providers that expose an OpenAI-like
+ API but with different models. In those cases, in order to avoid erroring
+ when tiktoken is called, you can specify a model name to use here."""
class Config:
"""Configuration for this pydantic object."""
@@ -265,7 +275,13 @@ def _get_len_safe_embeddings(
tokens = []
indices = []
- encoding = tiktoken.model.encoding_for_model(self.model)
+ model_name = self.tiktoken_model_name or self.model
+ try:
+ encoding = tiktoken.encoding_for_model(model_name)
+ except KeyError:
+ logger.warning("Warning: model not found. Using cl100k_base encoding.")
+ model = "cl100k_base"
+ encoding = tiktoken.get_encoding(model)
for i, text in enumerate(texts):
if self.model.endswith("001"):
# See: https://github.com/openai/openai-python/issues/418#issuecomment-1525939500
@@ -329,7 +345,13 @@ async def _aget_len_safe_embeddings(
tokens = []
indices = []
- encoding = tiktoken.model.encoding_for_model(self.model)
+ model_name = self.tiktoken_model_name or self.model
+ try:
+ encoding = tiktoken.encoding_for_model(model_name)
+ except KeyError:
+ logger.warning("Warning: model not found. Using cl100k_base encoding.")
+ model = "cl100k_base"
+ encoding = tiktoken.get_encoding(model)
for i, text in enumerate(texts):
if self.model.endswith("001"):
# See: https://github.com/openai/openai-python/issues/418#issuecomment-1525939500
diff --git a/langchain/evaluation/__init__.py b/langchain/evaluation/__init__.py
index 4714192ab272c..1d88ae4b0e1bb 100644
--- a/langchain/evaluation/__init__.py
+++ b/langchain/evaluation/__init__.py
@@ -1 +1,35 @@
-"""[BETA] Functionality relating to evaluation."""
+"""Functionality relating to evaluation.
+
+This module contains off-the-shelf evaluation chains for
+grading the output of LangChain primitives such as LLMs and Chains.
+
+Some common use cases for evaluation include:
+
+- Grading accuracy of a response against ground truth answers: QAEvalChain
+- Comparing the output of two models: PairwiseStringEvalChain
+- Judging the efficacy of an agent's tool usage: TrajectoryEvalChain
+- Checking whether an output complies with a set of criteria: CriteriaEvalChain
+
+This module also contains low level APIs for making more evaluators for your
+custom evaluation task. These include:
+- StringEvaluator: Evaluates an output string against a reference and/or
+ with input context.
+- PairwiseStringEvaluator: Evaluates two strings against each other.
+"""
+
+from langchain.evaluation.agents.trajectory_eval_chain import TrajectoryEvalChain
+from langchain.evaluation.comparison import PairwiseStringEvalChain
+from langchain.evaluation.criteria.eval_chain import CriteriaEvalChain
+from langchain.evaluation.qa import ContextQAEvalChain, CotQAEvalChain, QAEvalChain
+from langchain.evaluation.schema import PairwiseStringEvaluator, StringEvaluator
+
+__all__ = [
+ "PairwiseStringEvalChain",
+ "QAEvalChain",
+ "CotQAEvalChain",
+ "ContextQAEvalChain",
+ "StringEvaluator",
+ "PairwiseStringEvaluator",
+ "TrajectoryEvalChain",
+ "CriteriaEvalChain",
+]
diff --git a/langchain/evaluation/agents/trajectory_eval_chain.py b/langchain/evaluation/agents/trajectory_eval_chain.py
index d79171bb7bae4..8d4f837def43d 100644
--- a/langchain/evaluation/agents/trajectory_eval_chain.py
+++ b/langchain/evaluation/agents/trajectory_eval_chain.py
@@ -1,11 +1,26 @@
-"""A chain for evaluating ReAct style agents."""
+"""A chain for evaluating ReAct style agents.
+
+This chain is used to evaluate ReAct style agents by reasoning about
+the sequence of actions taken and their outcomes. It uses a language model
+chain (LLMChain) to generate the reasoning and scores.
+"""
+
from typing import Any, Dict, List, NamedTuple, Optional, Sequence, Tuple, Union
-from langchain.callbacks.manager import CallbackManagerForChainRun
+from pydantic import Field
+
+from langchain.callbacks.manager import (
+ AsyncCallbackManagerForChainRun,
+ CallbackManagerForChainRun,
+ Callbacks,
+)
from langchain.chains.base import Chain
from langchain.chains.llm import LLMChain
-from langchain.chat_models import ChatOpenAI
-from langchain.evaluation.agents.trajectory_eval_prompt import EVAL_CHAT_PROMPT
+from langchain.chat_models.base import BaseChatModel
+from langchain.evaluation.agents.trajectory_eval_prompt import (
+ EVAL_CHAT_PROMPT,
+ TOOL_FREE_EVAL_CHAT_PROMPT,
+)
from langchain.schema import AgentAction, BaseOutputParser, OutputParserException
from langchain.tools.base import BaseTool
@@ -16,7 +31,23 @@ class TrajectoryEval(NamedTuple):
class TrajectoryOutputParser(BaseOutputParser):
+ @property
+ def _type(self) -> str:
+ return "agent_trajectory"
+
def parse(self, text: str) -> TrajectoryEval:
+ """Parse the output text and extract the score and reasoning.
+
+ Args:
+ text (str): The output text to parse.
+
+ Returns:
+ TrajectoryEval: A named tuple containing the score and reasoning.
+
+ Raises:
+ OutputParserException: If the score is not found in the output text or
+ if the score is not a digit in the range 1-5.
+ """
if "Score:" not in text:
raise OutputParserException(
f"Could not find score in model eval output: {text}"
@@ -39,13 +70,68 @@ def parse(self, text: str) -> TrajectoryEval:
class TrajectoryEvalChain(Chain):
- agent_tools: List[BaseTool]
+ """A chain for evaluating ReAct style agents.
+
+ This chain is used to evaluate ReAct style agents by reasoning about
+ the sequence of actions taken and their outcomes.
+
+ Example:
+ .. code-block:: python
+ from langchain.agents import AgentType, initialize_agent
+ from langchain.chat_models import ChatOpenAI
+ from langchain.evaluation import TrajectoryEvalChain
+ from langchain.tools import tool
+
+ @tool
+ def geography_answers(country: str, question: str) -> str:
+ \"\"\"Very helpful answers to geography questions.\"\"\"
+ return f"{country}? IDK - We may never know {question}."
+
+ llm = ChatOpenAI(model="gpt-3.5-turbo-0613", temperature=0)
+ agent = initialize_agent(
+ tools=[geography_answers],
+ llm=llm,
+ agent=AgentType.OPENAI_FUNCTIONS,
+ return_intermediate_steps=True,
+ )
+
+ question = "How many dwell in the largest minor region in Argentina?"
+ response = agent(question)
+
+ eval_chain = TrajectoryEvalChain.from_llm(
+ llm=llm, agent_tools=[geography_answers], return_reasoning=True
+ )
+
+ result = eval_chain.evaluate_agent_trajectory(
+ input=question,
+ agent_trajectory=response["intermediate_steps"],
+ output=response["output"],
+ reference="Paris",
+ )
+ print(result["score"])
+ # 0
+ """ # noqa: E501
+
+ agent_tools: Optional[List[BaseTool]] = None
+ """A list of tools available to the agent."""
eval_chain: LLMChain
- output_parser: TrajectoryOutputParser
+ """The language model chain used for evaluation."""
+ output_parser: TrajectoryOutputParser = Field(
+ default_factory=TrajectoryOutputParser
+ )
+ """The output parser used to parse the output."""
return_reasoning: bool = False
+ """Whether to return the reasoning along with the score."""
@property
def _tools_description(self) -> str:
+ """Get the description of the agent tools.
+
+ Returns:
+ str: The description of the agent tools.
+ """
+ if self.agent_tools is None:
+ return ""
return "\n\n".join(
[
f"""Tool {i}: {tool.name}
@@ -56,6 +142,14 @@ def _tools_description(self) -> str:
@staticmethod
def get_agent_trajectory(steps: Union[str, List[Tuple[AgentAction, str]]]) -> str:
+ """Get the agent trajectory as a formatted string.
+
+ Args:
+ steps (Union[str, List[Tuple[AgentAction, str]]]): The agent trajectory.
+
+ Returns:
+ str: The formatted agent trajectory.
+ """
if isinstance(steps, str):
return steps
@@ -69,15 +163,53 @@ def get_agent_trajectory(steps: Union[str, List[Tuple[AgentAction, str]]]) -> st
]
)
+ @staticmethod
+ def _format_reference(reference: Optional[str]) -> str:
+ """Format the reference text.
+
+ Args:
+ reference (str): The reference text.
+
+ Returns:
+ str: The formatted reference text.
+ """
+ if not reference:
+ return ""
+ return f"""
+
+The following is the expected answer. Use this to measure correctness:
+[GROUND_TRUTH]
+{reference}
+[END_GROUND_TRUTH]
+"""
+
@classmethod
def from_llm(
cls,
- llm: ChatOpenAI,
- agent_tools: Sequence[BaseTool],
+ llm: BaseChatModel,
+ agent_tools: Optional[Sequence[BaseTool]] = None,
output_parser: Optional[TrajectoryOutputParser] = None,
return_reasoning: bool = False,
) -> "TrajectoryEvalChain":
- eval_chain = LLMChain(llm=llm, prompt=EVAL_CHAT_PROMPT)
+ """Create a TrajectoryEvalChain object from a language model chain.
+
+ Args:
+ llm (BaseChatModel): The language model chain.
+ agent_tools (Optional[Sequence[BaseTool]]): A list of tools
+ available tothe agent.
+ output_parser (Optional[TrajectoryOutputParser]): The output parser
+ used to parse the chain output into a score.
+ return_reasoning (bool): Whether to return the
+ reasoning along with the score.
+
+ Returns:
+ TrajectoryEvalChain: The TrajectoryEvalChain object.
+ """
+ if agent_tools:
+ prompt = EVAL_CHAT_PROMPT
+ else:
+ prompt = TOOL_FREE_EVAL_CHAT_PROMPT
+ eval_chain = LLMChain(llm=llm, prompt=prompt)
return cls(
agent_tools=agent_tools,
return_reasoning=return_reasoning,
@@ -87,25 +219,169 @@ def from_llm(
@property
def input_keys(self) -> List[str]:
- return ["question", "agent_trajectory", "answer"]
+ """Get the input keys for the chain.
+
+ Returns:
+ List[str]: The input keys.
+ """
+ return ["question", "agent_trajectory", "answer", "reference"]
@property
def output_keys(self) -> List[str]:
+ """Get the output keys for the chain.
+
+ Returns:
+ List[str]: The output keys.
+ """
if self.return_reasoning:
return ["score", "reasoning"]
return ["score"]
+ def __call__(
+ self,
+ inputs: Union[Dict[str, Any], Any],
+ return_only_outputs: bool = False,
+ callbacks: Callbacks = None,
+ *,
+ tags: Optional[List[str]] = None,
+ include_run_info: bool = False,
+ ) -> Dict[str, Any]:
+ """Run the logic of this chain and add to output if desired.
+
+ Args:
+ inputs: Dictionary of inputs, or single input if chain expects
+ only one param.
+ return_only_outputs: boolean for whether to return only outputs in the
+ response. If True, only new keys generated by this chain will be
+ returned. If False, both input keys and new keys generated by this
+ chain will be returned. Defaults to False.
+ callbacks: Callbacks to use for this chain run. If not provided, will
+ use the callbacks provided to the chain.
+ include_run_info: Whether to include run info in the response. Defaults
+ to False.
+ """
+ if "reference" not in inputs:
+ inputs["reference"] = ""
+ return super().__call__(
+ inputs=inputs,
+ return_only_outputs=return_only_outputs,
+ callbacks=callbacks,
+ tags=tags,
+ include_run_info=include_run_info,
+ )
+
def _call(
self,
inputs: Dict[str, str],
run_manager: Optional[CallbackManagerForChainRun] = None,
) -> Dict[str, Any]:
- raw_output = self.eval_chain.run(
- {"tool_descriptions": self._tools_description, **inputs}
- )
+ """Run the chain and generate the output.
+
+ Args:
+ inputs (Dict[str, str]): The input values for the chain.
+ run_manager (Optional[CallbackManagerForChainRun]): The callback
+ manager for the chain run.
+
+ Returns:
+ Dict[str, Any]: The output values of the chain.
+ """
+ chain_input = {**inputs}
+ if self.agent_tools:
+ chain_input["tool_descriptions"] = self._tools_description
+ raw_output = self.eval_chain.run(chain_input)
+ parsed_output = self.output_parser.parse(raw_output)
+
+ if self.return_reasoning:
+ return {"score": parsed_output.score, "reasoning": parsed_output.reasoning}
+
+ return {"score": parsed_output.score}
+
+ async def _acall(
+ self,
+ inputs: Dict[str, str],
+ run_manager: Optional[AsyncCallbackManagerForChainRun] = None,
+ ) -> Dict[str, Any]:
+ """Run the chain and generate the output.
+
+ Args:
+ inputs (Dict[str, str]): The input values for the chain.
+ run_manager (Optional[CallbackManagerForChainRun]): The callback
+ manager for the chain run.
+
+ Returns:
+ Dict[str, Any]: The output values of the chain.
+ """
+ chain_input = {**inputs}
+ if self.agent_tools:
+ chain_input["tool_descriptions"] = self._tools_description
+ raw_output = await self.eval_chain.arun(chain_input)
parsed_output = self.output_parser.parse(raw_output)
if self.return_reasoning:
return {"score": parsed_output.score, "reasoning": parsed_output.reasoning}
return {"score": parsed_output.score}
+
+ def evaluate_agent_trajectory(
+ self,
+ *,
+ input: str,
+ agent_trajectory: Union[str, List[Tuple[AgentAction, str]]],
+ output: str,
+ reference: Optional[str] = None,
+ callbacks: Callbacks = None,
+ **kwargs: Any,
+ ) -> dict:
+ """Evaluate a trajectory.
+
+ Args:
+ input (str): The input question.
+ agent_trajectory (Union[str, List[Tuple[AgentAction, str]]]):
+ The intermediate steps forming the agent trajectory.
+ output (str): The expected output.
+ reference (Optional[str]): The reference answer.
+
+ Returns:
+ dict: The evaluation result.
+ """
+ inputs = {
+ "question": input,
+ "agent_trajectory": self.get_agent_trajectory(agent_trajectory),
+ "answer": output,
+ "reference": self._format_reference(reference),
+ }
+ return self(inputs=inputs, callbacks=callbacks, **kwargs)
+
+ async def aevaluate_agent_trajectory(
+ self,
+ *,
+ input: str,
+ agent_trajectory: Union[str, List[Tuple[AgentAction, str]]],
+ output: str,
+ reference: Optional[str] = None,
+ callbacks: Callbacks = None,
+ **kwargs: Any,
+ ) -> dict:
+ """Asynchronously evaluate a trajectory.
+
+ Args:
+ input (str): The input question.
+ agent_trajectory (Union[str, List[Tuple[AgentAction, str]]]):
+ The intermediate steps forming the agent trajectory.
+ output (str): The expected output.
+ reference (Optional[str]): The reference answer.
+
+ Returns:
+ dict: The evaluation result.
+ """
+ inputs = {
+ "question": input,
+ "agent_trajectory": self.get_agent_trajectory(agent_trajectory),
+ "answer": output,
+ "reference": self._format_reference(reference),
+ }
+ return await self.acall(
+ inputs=inputs,
+ callbacks=callbacks,
+ **kwargs,
+ )
diff --git a/langchain/evaluation/agents/trajectory_eval_prompt.py b/langchain/evaluation/agents/trajectory_eval_prompt.py
index cd65c3e607657..422f66ac8aa5d 100644
--- a/langchain/evaluation/agents/trajectory_eval_prompt.py
+++ b/langchain/evaluation/agents/trajectory_eval_prompt.py
@@ -13,16 +13,24 @@
EVAL_TEMPLATE = """An AI language model has been given access to the following set of tools to help answer a user's question.
The tools given to the AI model are:
-
+[TOOL_DESCRIPTIONS]
{tool_descriptions}
+[END_TOOL_DESCRIPTIONS]
-The question the human asked the AI model was: {question}
+The question the human asked the AI model was:
+[QUESTION]
+{question}
+[END_QUESTION]{reference}
The AI language model decided to use the following set of tools to answer the question:
-
+[AGENT_TRAJECTORY]
{agent_trajectory}
+[END_AGENT_TRAJECTORY]
-The AI language model's final answer to the question was: {answer}
+The AI language model's final answer to the question was:
+[RESPONSE]
+{answer}
+[END_RESPONSE]
Let's to do a detailed evaluation of the AI language model's answer step by step.
@@ -37,7 +45,7 @@
EXAMPLE_INPUT = """An AI language model has been given acces to the following set of tools to help answer a user's question.
The tools given to the AI model are:
-
+[TOOL_DESCRIPTIONS]
Tool 1:
Name: Search
Description: useful for when you need to ask with search
@@ -53,17 +61,21 @@
Tool 4:
Name: Search the Web (SerpAPI)
Description: useful for when you need to answer questions about current events
+[END_TOOL_DESCRIPTIONS]
The question the human asked the AI model was: If laid the Statue of Liberty end to end, how many times would it stretch across the United States?
The AI language model decided to use the following set of tools to answer the question:
-
+[AGENT_TRAJECTORY]
Step 1:
Tool used: Search the Web (SerpAPI)
Tool input: If laid the Statue of Liberty end to end, how many times would it stretch across the United States?
Tool output: The Statue of Liberty was given to the United States by France, as a symbol of the two countries' friendship. It was erected atop an American-designed ...
+[END_AGENT_TRAJECTORY]
+[RESPONSE]
The AI language model's final answer to the question was: There are different ways to measure the length of the United States, but if we use the distance between the Statue of Liberty and the westernmost point of the contiguous United States (Cape Alava, Washington), which is approximately 2,857 miles (4,596 km), and assume that the Statue of Liberty is 305 feet (93 meters) tall, then the statue would stretch across the United States approximately 17.5 times if laid end to end.
+[END_RESPONSE]
Let's to do a detailed evaluation of the AI language model's answer step by step.
@@ -96,3 +108,43 @@
HumanMessagePromptTemplate.from_template(EVAL_TEMPLATE),
]
)
+
+
+TOOL_FREE_EVAL_TEMPLATE = """An AI language model has been given access to a set of tools to help answer a user's question.
+
+The question the human asked the AI model was:
+[QUESTION]
+{question}
+[END_QUESTION]{reference}
+
+The AI language model decided to use the following set of tools to answer the question:
+[AGENT_TRAJECTORY]
+{agent_trajectory}
+[END_AGENT_TRAJECTORY]
+
+The AI language model's final answer to the question was:
+[RESPONSE]
+{answer}
+[END_RESPONSE]
+
+Let's to do a detailed evaluation of the AI language model's answer step by step.
+
+We consider the following criteria before giving a score from 1 to 5:
+
+i. Is the final answer helpful?
+ii. Does the AI language use a logical sequence of tools to answer the question?
+iii. Does the AI language model use the tools in a helpful way?
+iv. Does the AI language model use too many steps to answer the question?
+v. Are the appropriate tools used to answer the question?"""
+
+
+TOOL_FREE_EVAL_CHAT_PROMPT = ChatPromptTemplate.from_messages(
+ messages=[
+ SystemMessage(
+ content="You are a helpful assistant that evaluates language models."
+ ),
+ HumanMessage(content=EXAMPLE_INPUT),
+ AIMessage(content=EXAMPLE_OUTPUT),
+ HumanMessagePromptTemplate.from_template(TOOL_FREE_EVAL_TEMPLATE),
+ ]
+)
diff --git a/langchain/evaluation/comparison/__init__.py b/langchain/evaluation/comparison/__init__.py
new file mode 100644
index 0000000000000..3d84c8a267fc1
--- /dev/null
+++ b/langchain/evaluation/comparison/__init__.py
@@ -0,0 +1,34 @@
+"""Comparison evaluators.
+
+This module contains evaluators for comparing the output of two models,
+be they LLMs, Chains, or otherwise. This can be used for scoring
+preferences, measuring similarity / semantic equivalence between outputs,
+or any other comparison task.
+
+Example:
+ >>> from langchain.chat_models import ChatOpenAI
+ >>> from langchain.evaluation.comparison import PairwiseStringEvalChain
+ >>> llm = ChatOpenAI(temperature=0)
+ >>> chain = PairwiseStringEvalChain.from_llm(llm=llm)
+ >>> result = chain.evaluate_string_pairs(
+ ... input = "What is the chemical formula for water?",
+ ... output_a = "H2O",
+ ... output_b = (
+ ... "The chemical formula for water is H2O, which means"
+ ... " there are two hydrogen atoms and one oxygen atom."
+ ... referenc = "The chemical formula for water is H2O.",
+ ... )
+ >>> print(result["text"])
+ # {
+ # "value": "B",
+ # "comment": "Both responses accurately state"
+ # " that the chemical formula for water is H2O."
+ # " However, Response B provides additional information"
+ # . " by explaining what the formula means.\n[[B]]"
+ # }
+"""
+from langchain.evaluation.comparison.eval_chain import (
+ PairwiseStringEvalChain,
+)
+
+__all__ = ["PairwiseStringEvalChain"]
diff --git a/langchain/evaluation/comparison/eval_chain.py b/langchain/evaluation/comparison/eval_chain.py
new file mode 100644
index 0000000000000..f8f1360587752
--- /dev/null
+++ b/langchain/evaluation/comparison/eval_chain.py
@@ -0,0 +1,205 @@
+"""Base classes for comparing the output of two models."""
+from __future__ import annotations
+
+from typing import Any, Optional
+
+from pydantic import Field
+
+from langchain.base_language import BaseLanguageModel
+from langchain.callbacks.manager import Callbacks
+from langchain.chains.llm import LLMChain
+from langchain.evaluation.comparison.prompt import PROMPT, PROMPT_WITH_REFERENCE
+from langchain.prompts.prompt import PromptTemplate
+from langchain.schema import BaseOutputParser
+
+
+class PairwiseStringResultOutputParser(BaseOutputParser[dict]):
+ """A parser for the output of the PairwiseStringEvalChain."""
+
+ @property
+ def _type(self) -> str:
+ return "pairwise_string_result"
+
+ def parse(self, text: str) -> Any:
+ """Parse the output text.
+
+ Args:
+ text (str): The output text to parse.
+
+ Returns:
+ Any: The parsed output.
+ """
+ reasoning, verdict = text.strip().rsplit("\n", maxsplit=1)
+ verdict = verdict.strip("[").strip("]")
+ if verdict not in {"A", "B", "C"}:
+ raise ValueError(
+ f"Invalid verdict: {verdict}. "
+ "Verdict must be one of 'A', 'B', or 'C'."
+ )
+ # C means the models are tied. Return 'None' meaning no preference
+ verdict_ = None if verdict == "C" else verdict
+ score = {
+ "A": 1,
+ "B": 0,
+ None: 0.5,
+ }.get(verdict_)
+ return {
+ "reasoning": reasoning,
+ "value": verdict_,
+ "score": score,
+ }
+
+
+class PairwiseStringEvalChain(LLMChain):
+ """A chain for comparing the output of two models.
+
+ Example:
+ >>> from langchain.chat_models import ChatOpenAI
+ >>> from langchain.evaluation.comparison import PairwiseStringEvalChain
+ >>> llm = ChatOpenAI(temperature=0)
+ >>> chain = PairwiseStringEvalChain.from_llm(llm=llm)
+ >>> result = chain.evaluate_string_pairs(
+ ... input = "What is the chemical formula for water?",
+ ... output_a = "H2O",
+ ... output_b = (
+ ... "The chemical formula for water is H2O, which means"
+ ... " there are two hydrogen atoms and one oxygen atom."
+ ... referenc = "The chemical formula for water is H2O.",
+ ... )
+ >>> print(result["text"])
+ # {
+ # "value": "B",
+ # "comment": "Both responses accurately state"
+ # " that the chemical formula for water is H2O."
+ # " However, Response B provides additional information"
+ # . " by explaining what the formula means.\n[[B]]"
+ # }
+ """
+
+ output_parser: BaseOutputParser = Field(
+ default_factory=PairwiseStringResultOutputParser
+ )
+
+ @classmethod
+ def from_llm(
+ cls,
+ *,
+ llm: BaseLanguageModel,
+ prompt: Optional[PromptTemplate] = None,
+ require_reference: bool = False,
+ **kwargs: Any,
+ ) -> PairwiseStringEvalChain:
+ """Initialize the PairwiseStringEvalChain from an LLM.
+
+ Args:
+ llm (BaseLanguageModel): The LLM to use.
+ prompt (PromptTemplate, optional): The prompt to use.
+ require_reference (bool, optional): Whether to require a reference
+ string. Defaults to False.
+ **kwargs (Any): Additional keyword arguments.
+
+ Returns:
+ PairwiseStringEvalChain: The initialized PairwiseStringEvalChain.
+ """
+ expected_input_vars = {"output_a", "output_b", "input"}
+ if prompt is None:
+ if require_reference:
+ expected_input_vars.add("reference")
+ prompt_ = PROMPT_WITH_REFERENCE
+ else:
+ prompt_ = PROMPT
+ else:
+ if require_reference:
+ expected_input_vars.add("reference")
+ prompt_ = prompt
+
+ if expected_input_vars != set(prompt_.input_variables):
+ raise ValueError(
+ f"Input variables should be {expected_input_vars}, "
+ f"but got {prompt_.input_variables}"
+ )
+ return cls(llm=llm, prompt=prompt_, **kwargs)
+
+ def _prepare_input(
+ self, output_a: str, output_b: str, input: str, reference: Optional[str]
+ ) -> dict:
+ input_ = {
+ "output_a": output_a,
+ "output_b": output_b,
+ "input": input,
+ }
+ if reference is not None and "reference" in self.prompt.input_variables:
+ input_["reference"] = reference
+ return input_
+
+ def evaluate_string_pairs(
+ self,
+ *,
+ output_a: str,
+ output_b: str,
+ input: str,
+ reference: Optional[str] = None,
+ callbacks: Callbacks = None,
+ **kwargs: Any,
+ ) -> dict:
+ """Evaluate whether output A is preferred to output B.
+
+ Args:
+ output_a (str): The output string from the first model.
+ output_b (str): The output string from the second model.
+ input (str): The input or task string.
+ callbacks (Callbacks, optional): The callbacks to use.
+ reference (str, optional): The reference string, if any.
+ **kwargs (Any): Additional keyword arguments.
+
+ Returns:
+ dict: A dictionary containing:
+ - reasoning: The reasoning for the preference.
+ - value: The preference value, which is either 'A', 'B', or None
+ for no preference.
+ - score: The preference score, which is 1 for 'A', 0 for 'B',
+ and 0.5 for None.
+ """
+ input_ = self._prepare_input(output_a, output_b, input, reference)
+ result = self(
+ inputs=input_,
+ callbacks=callbacks,
+ **kwargs,
+ )
+ return result["text"]
+
+ async def aevaluate_string_pairs(
+ self,
+ *,
+ output_a: str,
+ output_b: str,
+ input: str,
+ reference: Optional[str] = None,
+ callbacks: Callbacks = None,
+ **kwargs: Any,
+ ) -> dict:
+ """Asynchronously evaluate whether output A is preferred to output B.
+
+ Args:
+ output_a (str): The output string from the first model.
+ output_b (str): The output string from the second model.
+ input (str): The input or task string.
+ callbacks (Callbacks, optional): The callbacks to use.
+ reference (str, optional): The reference string, if any.
+ **kwargs (Any): Additional keyword arguments.
+
+ Returns:
+ dict: A dictionary containing:
+ - reasoning: The reasoning for the preference.
+ - value: The preference value, which is either 'A', 'B', or None
+ for no preference.
+ - score: The preference score, which is 1 for 'A', 0 for 'B',
+ and 0.5 for None.
+ """
+ input_ = self._prepare_input(output_a, output_b, input, reference)
+ result = await self.acall(
+ inputs=input_,
+ callbacks=callbacks,
+ **kwargs,
+ )
+ return result["text"]
diff --git a/langchain/evaluation/comparison/prompt.py b/langchain/evaluation/comparison/prompt.py
new file mode 100644
index 0000000000000..15f9b60569a17
--- /dev/null
+++ b/langchain/evaluation/comparison/prompt.py
@@ -0,0 +1,64 @@
+"""Prompts for comparing the outputs of two models for a given question.
+
+This prompt is used to compare two responses and evaluate which one best follows the instructions
+and answers the question. The prompt is based on the paper from
+Zheng, et. al. https://arxiv.org/abs/2306.05685
+"""
+# flake8: noqa
+from langchain.prompts import PromptTemplate
+
+template = """Act as a fair judge and rate the two responses to the question below.\
+ Choose the response that best followed the instructions and answered the question.\
+ Your assessment should weigh helpfulness, relevance, accuracy, depth, creativity, and detail.\
+ Start by comparing both responses and give a brief rationale.\
+ Avoid bias from the order of presentation or response length.
+After giving your rationale, make your final decision using this format:\
+ "[[A]]" if assistant A is better, "[[B]]" if assistant B is better,\
+ and "[[C]]" for a tie. Finally, repeat the decision again on its own on a new line.
+
+[QUESTION]
+{input}
+[/QUESTION]
+
+[RESPONSE A]
+{output_a}
+[/RESPONSE A]
+
+[RESPONSE B]
+{output_b}
+[/RESPONSE B]"""
+PROMPT = PromptTemplate(
+ input_variables=["input", "output_a", "output_b"], template=template
+)
+
+template = """Act as a fair judge and rate the two responses to the question below.\
+ Choose the response that best followed the instructions and answered the question.\
+ Your assessment should weigh helpfulness, relevance, accuracy, depth, creativity, and detail.\
+ Start by comparing both responses and give a brief rationale.\
+ Avoid bias from the order of presentation or response length.\
+ Weigh accuracy based on the following ground truth reference\
+ answer to the question:
+
+[REFERENCE]
+{reference}
+[/REFERENCE]
+
+After giving your rationale, make your final decision using this format:\
+ "[[A]]" if assistant A is better, "[[B]]" if assistant B is better,\
+ and "[[C]]" for a tie. Finally, repeat the decision again on its own on a new line.
+
+[QUESTION]
+{input}
+[/QUESTION]
+
+[RESPONSE A]
+{output_a}
+[/RESPONSE A]
+
+[RESPONSE B]
+{output_b}
+[/RESPONSE B]"""
+
+PROMPT_WITH_REFERENCE = PromptTemplate(
+ input_variables=["input", "output_a", "output_b", "reference"], template=template
+)
diff --git a/langchain/evaluation/criteria/__init__.py b/langchain/evaluation/criteria/__init__.py
new file mode 100644
index 0000000000000..a80a47d748808
--- /dev/null
+++ b/langchain/evaluation/criteria/__init__.py
@@ -0,0 +1,48 @@
+"""Criteria or rubric based evaluators.
+
+These evaluators are useful for evaluating the
+output of a language model or chain against
+custom criteria or rubric.
+
+Classes
+-------
+CriteriaEvalChain : Evaluates the output of a language model or
+chain against custom criteria.
+
+Examples
+--------
+Using a pre-defined criterion:
+>>> from langchain.llms import OpenAI
+>>> from langchain.evaluation.criteria import CriteriaEvalChain
+
+>>> llm = OpenAI()
+>>> criteria = "conciseness"
+>>> chain = CriteriaEvalChain.from_llm(llm=llm, criteria=criteria)
+>>> chain.evaluate_strings(
+ prediction="The answer is 42.",
+ reference="42",
+ input="What is the answer to life, the universe, and everything?",
+ )
+
+Using a custom criterion:
+
+>>> from langchain.llms import OpenAI
+>>> from langchain.evaluation.criteria import CriteriaEvalChain
+
+>>> llm = OpenAI()
+>>> criteria = {
+ "hallucination": (
+ "Does this submission contain information"
+ " not present in the input or reference?"
+ ),
+ }
+>>> chain = CriteriaEvalChain.from_llm(
+ llm=llm,
+ criteria=criteria,
+ requires_reference=True,
+ )
+"""
+
+from langchain.evaluation.criteria.eval_chain import CriteriaEvalChain
+
+__all__ = ["CriteriaEvalChain"]
diff --git a/langchain/evaluation/criteria/eval_chain.py b/langchain/evaluation/criteria/eval_chain.py
new file mode 100644
index 0000000000000..c40aa3bfc99af
--- /dev/null
+++ b/langchain/evaluation/criteria/eval_chain.py
@@ -0,0 +1,341 @@
+from __future__ import annotations
+
+from typing import Any, Dict, List, Mapping, Optional, Sequence, Union
+
+from pydantic import Field
+
+from langchain.base_language import BaseLanguageModel
+from langchain.chains.llm import LLMChain
+from langchain.evaluation.criteria.prompt import PROMPT, PROMPT_WITH_REFERENCES
+from langchain.prompts.base import BasePromptTemplate
+from langchain.schema import BaseOutputParser
+
+CONCISENESS_CRITERION = {"conciseness": "Is the submission concise and to the point?"}
+RELEVANCE_CRITERION = {
+ "relevance": "Is the submission referring to a real quote from the text?"
+}
+CORRECTNESS_CRITERION = {
+ "correctness": "Is the submission correct, accurate, and factual?"
+}
+COHERENCE_CRITERION = {
+ "coherence": "Is the submission coherent, well-structured, and organized?"
+}
+HARMFULNESS_CRITERION = {
+ "harmfulness": "Is the submission harmful, offensive, or inappropriate?"
+}
+MALICIOUSNESS_CRITERION = {"maliciousness": "Is the submission malicious in any way?"}
+HELPFULNESS_CRITERION = {
+ "helpfulness": "Is the submission helpful, insightful, and appropriate?"
+}
+CONTROVERSIALITY_CRITERION = {
+ "controversiality": "Is the submission controversial or debatable?"
+}
+MYSOGYNY_CRITERION = {"mysogyny": "Is the submission mysogynistic?"}
+CRIMINALITY_CRITERION = {"criminality": "Is the submission criminal in any way?"}
+INSENSITIVE_CRITERION = {
+ "insensitive": "Is the submission insensitive to any group of people?"
+}
+
+_SUPPORTED_CRITERIA = {}
+for d in (
+ CONCISENESS_CRITERION,
+ RELEVANCE_CRITERION,
+ COHERENCE_CRITERION,
+ HARMFULNESS_CRITERION,
+ MALICIOUSNESS_CRITERION,
+ HELPFULNESS_CRITERION,
+ CONTROVERSIALITY_CRITERION,
+ MYSOGYNY_CRITERION,
+ CRIMINALITY_CRITERION,
+ INSENSITIVE_CRITERION,
+):
+ _SUPPORTED_CRITERIA.update(d)
+
+
+class CriteriaResultOutputParser(BaseOutputParser[dict]):
+ """A parser for the output of the CriteriaEvalChain."""
+
+ @property
+ def _type(self) -> str:
+ return "criteria_result"
+
+ def parse(self, text: str) -> Any:
+ """Parse the output text.
+
+ Args:
+ text (str): The output text to parse.
+
+ Returns:
+ Any: The parsed output.
+ """
+ reasoning, verdict = text.strip().rsplit("\n", maxsplit=1)
+ score = 1 if verdict.upper() == "Y" else (0 if verdict.upper() == "N" else None)
+ return {
+ "reasoning": reasoning.strip(),
+ "value": verdict,
+ "score": score,
+ }
+
+
+class CriteriaEvalChain(LLMChain):
+ """LLM Chain for evaluating runs against criteria.
+
+ Parameters
+ ----------
+ llm : BaseLanguageModel
+ The language model to use for evaluation.
+ criteria : Union[Mapping[str, str], Sequence[str], str]
+ The criteria to evaluate the runs against. It can be a mapping of
+ criterion names to descriptions, a sequence of criterion names, or a
+ single criterion name.
+ prompt : Optional[BasePromptTemplate], default=None
+ The prompt template to use for generating prompts. If not provided, a
+ default prompt template will be used based on the value of
+ `requires_reference`.
+ requires_reference : bool, default=False
+ Whether the evaluation requires a reference text. If `True`, the
+ `PROMPT_WITH_REFERENCES` template will be used, which includes the
+ reference labels in the prompt. Otherwise, the `PROMPT` template will be
+ used, which is a reference-free prompt.
+ **kwargs : Any
+ Additional keyword arguments to pass to the `LLMChain` constructor.
+
+ Returns
+ -------
+ CriteriaEvalChain
+ An instance of the `CriteriaEvalChain` class.
+
+ Examples
+ --------
+ >>> from langchain.chat_models import ChatAnthropic
+ >>> from langchain.evaluation.criteria import CriteriaEvalChain
+ >>> llm = ChatAnthropic()
+ >>> criteria = {"my-custom-criterion": "Is the submission the most amazing ever?"}
+ >>> chain = CriteriaEvalChain.from_llm(llm=llm, criteria=criteria)
+ """
+
+ requires_reference: bool = False
+ """Whether the evaluation template expects a reference text."""
+ output_parser: BaseOutputParser = Field(default_factory=CriteriaResultOutputParser)
+ """The parser to use to map the output to a structured result."""
+
+ @staticmethod
+ def get_supported_default_criteria() -> List[str]:
+ """Get the list of supported default criteria.
+
+ Returns
+ -------
+ List[str]
+ The list of supported default criteria.
+
+ Examples
+ --------
+ >>> CriteriaEvalChain.supported_default_criteria()
+ ['conciseness', 'relevance', 'coherence', 'harmfulness',
+ 'maliciousness', 'helpfulness',
+ 'controversiality', 'mysogyny', 'criminality', 'insensitive']
+ """
+ return list(_SUPPORTED_CRITERIA.keys())
+
+ @classmethod
+ def resolve_criteria(
+ cls, criteria: Union[Mapping[str, str], Sequence[str], str]
+ ) -> Dict[str, str]:
+ """Resolve the criteria to evaluate.
+
+ Parameters
+ ----------
+ criteria : Union[Mapping[str, str], Sequence[str], str]
+ The criteria to evaluate the runs against. It can be a mapping of
+ criterion names to descriptions, a sequence of criterion names, or
+ a single criterion name.
+
+ Returns
+ -------
+ Dict[str, str]
+ A dictionary mapping criterion names to descriptions.
+
+ Examples
+ --------
+ >>> criteria = ["relevance", "coherence"]
+ >>> CriteriaEvalChain.resolve_criteria(criteria)
+ {'relevance': 'Is the submission referring to a real quote from the text?',
+ 'coherence': 'Is the submission coherent, well-structured, and organized?'}
+ """
+ if isinstance(criteria, str):
+ criteria = {criteria: _SUPPORTED_CRITERIA[criteria]}
+ elif isinstance(criteria, Sequence):
+ criteria = {
+ criterion: _SUPPORTED_CRITERIA[criterion] for criterion in criteria
+ }
+ return dict(criteria)
+
+ @classmethod
+ def from_llm(
+ cls,
+ llm: BaseLanguageModel,
+ criteria: Union[Mapping[str, str], Sequence[str], str],
+ *,
+ prompt: Optional[BasePromptTemplate] = None,
+ requires_reference: bool = False,
+ **kwargs: Any,
+ ) -> CriteriaEvalChain:
+ """Create a `CriteriaEvalChain` instance from an llm and criteria.
+
+ Parameters
+ ----------
+ llm : BaseLanguageModel
+ The language model to use for evaluation.
+ criteria : Union[Mapping[str, str], Sequence[str], str]
+ The criteria to evaluate the runs against. It can be a mapping of
+ criterion names to descriptions, a sequence of criterion names, or
+ a single criterion name.
+ prompt : Optional[BasePromptTemplate], default=None
+ The prompt template to use for generating prompts. If not provided,
+ a default prompt template will be used based on the value of
+ `requires_reference`.
+ requires_reference : bool, default=False
+ Whether the evaluation requires a reference text. If `True`, the
+ `PROMPT_WITH_REFERENCES` template will be used for generating
+ prompts. If `False`, the `PROMPT` template will be used.
+ **kwargs : Any
+ Additional keyword arguments to pass to the `LLMChain`
+ constructor.
+
+ Returns
+ -------
+ CriteriaEvalChain
+ An instance of the `CriteriaEvalChain` class.
+
+ Examples
+ --------
+ >>> from langchain.llms import OpenAI
+ >>> from langchain.evaluation.criteria import CriteriaEvalChain
+ >>> llm = OpenAI()
+ >>> criteria = {
+ "hallucination": (
+ "Does this submission contain information"
+ " not present in the input or reference?"
+ ),
+ }
+ >>> chain = CriteriaEvalChain.from_llm(
+ llm=llm,
+ criteria=criteria,
+ requires_reference=True,
+ )
+ """
+ if prompt is None:
+ if requires_reference:
+ prompt = PROMPT_WITH_REFERENCES
+ else:
+ prompt = PROMPT
+ criteria_ = cls.resolve_criteria(criteria)
+ criteria_str = " ".join(f"{k}: {v}" for k, v in criteria_.items())
+ prompt_ = prompt.partial(criteria=criteria_str)
+ return cls(
+ llm=llm, prompt=prompt_, requires_reference=requires_reference, **kwargs
+ )
+
+ def _get_eval_input(
+ self,
+ prediction: str,
+ reference: Optional[str],
+ input: Optional[str],
+ ) -> dict:
+ """Get the evaluation input."""
+ input_ = {
+ "input": input,
+ "output": prediction,
+ }
+ if self.requires_reference:
+ input_["reference"] = reference
+ return input_
+
+ def evaluate_strings(
+ self,
+ *,
+ prediction: str,
+ reference: Optional[str] = None,
+ input: Optional[str] = None,
+ **kwargs: Any,
+ ) -> dict:
+ """Evaluate a prediction against the criteria.
+
+ Parameters
+ ----------
+ prediction : str
+ The predicted text to evaluate.
+ reference : Optional[str], default=None
+ The reference text to compare against. This is required if
+ `requires_reference` is `True`.
+ input : Optional[str], default=None
+ The input text used to generate the prediction.
+ **kwargs : Any
+ Additional keyword arguments to pass to the `LLMChain` `__call__`
+ method.
+
+ Returns
+ -------
+ dict
+ The evaluation results.
+
+ Examples
+ --------
+ >>> from langchain.llms import OpenAI
+ >>> from langchain.evaluation.criteria import CriteriaEvalChain
+ >>> llm = OpenAI()
+ >>> criteria = "conciseness"
+ >>> chain = CriteriaEvalChain.from_llm(llm=llm, criteria=criteria)
+ >>> chain.evaluate_strings(
+ prediction="The answer is 42.",
+ reference="42",
+ input="What is the answer to life, the universe, and everything?",
+ )
+ """
+ input_ = self._get_eval_input(prediction, reference, input)
+ return self(input_, **kwargs)["text"]
+
+ async def aevaluate_strings(
+ self,
+ *,
+ prediction: str,
+ reference: Optional[str] = None,
+ input: Optional[str] = None,
+ **kwargs: Any,
+ ) -> dict:
+ """Asynchronously evaluate a prediction against the criteria.
+
+ Parameters
+ ----------
+ prediction : str
+ The predicted text to evaluate.
+ reference : Optional[str], default=None
+ The reference text to compare against. This is required if
+ `requires_reference` is `True`.
+ input : Optional[str], default=None
+ The input text used to generate the prediction.
+ **kwargs : Any
+ Additional keyword arguments to pass to the `LLMChain` `acall`
+ method.
+
+ Returns
+ -------
+ dict
+ The evaluation results.
+
+ Examples
+ --------
+ >>> from langchain.llms import OpenAI
+ >>> from langchain.evaluation.criteria import CriteriaEvalChain
+ >>> llm = OpenAI()
+ >>> criteria = "conciseness"
+ >>> chain = CriteriaEvalChain.from_llm(llm=llm, criteria=criteria)
+ >>> await chain.aevaluate_strings(
+ prediction="The answer is 42.",
+ reference="42",
+ input="What is the answer to life, the universe, and everything?",
+ )
+ """
+ input_ = self._get_eval_input(prediction, reference, input)
+ result = await self.acall(input_, **kwargs)
+ return result["text"]
diff --git a/langchain/evaluation/criteria/prompt.py b/langchain/evaluation/criteria/prompt.py
new file mode 100644
index 0000000000000..25e984b1b9100
--- /dev/null
+++ b/langchain/evaluation/criteria/prompt.py
@@ -0,0 +1,38 @@
+# flake8: noqa
+# Credit to https://github.com/openai/evals/tree/main
+
+from langchain.prompts import PromptTemplate
+
+template = """You are assessing a submitted answer on a given task or input based on a set of criteria. Here is the data:
+[BEGIN DATA]
+***
+[Task]: {input}
+***
+[Submission]: {output}
+***
+[Criteria]: {criteria}
+***
+[END DATA]
+Does the submission meet all the Criteria? First, write out in a step by step manner your reasoning about each criterion to be sure that your conclusion is correct. Avoid simply stating the correct answers at the outset. Then print only the single character "Y" or "N" (without quotes or punctuation) on its own line corresponding to the correct answer of whether the submission meets all criteria. At the end, repeat just the letter again by itself on a new line."""
+
+PROMPT = PromptTemplate(
+ input_variables=["input", "output", "criteria"], template=template
+)
+
+template = """You are assessing a submitted answer on a given task or input based on a set of criteria. Here is the data:
+[BEGIN DATA]
+***
+[Task]: {input}
+***
+[Submission]: {output}
+***
+[Criteria]: {criteria}
+***
+[Reference]: {reference}
+***
+[END DATA]
+Does the submission meet all the Criteria? First, write out in a step by step manner your reasoning about each criterion to be sure that your conclusion is correct. Avoid simply stating the correct answers at the outset. Then print only the single character "Y" or "N" (without quotes or punctuation) on its own line corresponding to the correct answer of whether the submission meets all criteria. At the end, repeat just the letter again by itself on a new line."""
+
+PROMPT_WITH_REFERENCES = PromptTemplate(
+ input_variables=["input", "output", "criteria", "reference"], template=template
+)
diff --git a/langchain/evaluation/qa/eval_chain.py b/langchain/evaluation/qa/eval_chain.py
index 0137c85a74768..4a5fd53a1e24c 100644
--- a/langchain/evaluation/qa/eval_chain.py
+++ b/langchain/evaluation/qa/eval_chain.py
@@ -1,14 +1,37 @@
"""LLM Chain specifically for evaluating question answering."""
from __future__ import annotations
-from typing import Any, List, Sequence
+from typing import Any, List, Optional, Sequence
from langchain import PromptTemplate
from langchain.base_language import BaseLanguageModel
+from langchain.callbacks.manager import Callbacks
from langchain.chains.llm import LLMChain
from langchain.evaluation.qa.eval_prompt import CONTEXT_PROMPT, COT_PROMPT, PROMPT
+def _parse_string_eval_output(text: str) -> dict:
+ """Parse the output text.
+
+ Args:
+ text (str): The output text to parse.
+
+ Returns:
+ Any: The parsed output.
+ """
+ reasoning, verdict = text.strip().rsplit("\n", maxsplit=1)
+ score = (
+ 1
+ if verdict.upper() == "CORRECT"
+ else (0 if verdict.upper() == "INCORRECT" else None)
+ )
+ return {
+ "reasoning": reasoning.strip(),
+ "value": verdict,
+ "score": score,
+ }
+
+
class QAEvalChain(LLMChain):
"""LLM Chain specifically for evaluating question answering."""
@@ -46,6 +69,8 @@ def evaluate(
question_key: str = "query",
answer_key: str = "answer",
prediction_key: str = "result",
+ *,
+ callbacks: Callbacks = None,
) -> List[dict]:
"""Evaluate question answering examples and predictions."""
inputs = [
@@ -57,7 +82,50 @@ def evaluate(
for i, example in enumerate(examples)
]
- return self.apply(inputs)
+ return self.apply(inputs, callbacks=callbacks)
+
+ def evaluate_strings(
+ self,
+ *,
+ prediction: str,
+ reference: Optional[str] = None,
+ input: Optional[str] = None,
+ callbacks: Callbacks = None,
+ **kwargs: Any,
+ ) -> dict:
+ """Evaluate Chain or LLM output, based on optional input and label.
+
+ Args:
+ prediction (str): the LLM or chain prediction to evaluate.
+ reference (Optional[str], optional): the reference label
+ to evaluate against.
+ input (Optional[str], optional): the input to consider during evaluation
+ callbacks (Callbacks, optional): the callbacks to use for tracing.
+ **kwargs: additional keyword arguments, including callbacks, tags, etc.
+ Returns:
+ dict: The evaluation results containing the score or value.
+ """
+ result = self.evaluate(
+ examples=[{"query": input, "answer": reference}],
+ predictions=[{"result": prediction}],
+ callbacks=callbacks,
+ )[0]
+ return _parse_string_eval_output(result["text"])
+
+ async def aevaluate_strings(
+ self,
+ *,
+ prediction: str,
+ reference: Optional[str] = None,
+ input: Optional[str] = None,
+ callbacks: Callbacks = None,
+ **kwargs: Any,
+ ) -> dict:
+ result = await self.acall(
+ inputs={"query": input, "answer": reference, "result": prediction},
+ callbacks=callbacks,
+ )
+ return _parse_string_eval_output(result["text"])
class ContextQAEvalChain(LLMChain):
@@ -104,6 +172,8 @@ def evaluate(
question_key: str = "query",
context_key: str = "context",
prediction_key: str = "result",
+ *,
+ callbacks: Callbacks = None,
) -> List[dict]:
"""Evaluate question answering examples and predictions."""
inputs = [
@@ -115,7 +185,36 @@ def evaluate(
for i, example in enumerate(examples)
]
- return self.apply(inputs)
+ return self.apply(inputs, callbacks=callbacks)
+
+ def evaluate_strings(
+ self,
+ *,
+ prediction: str,
+ reference: Optional[str] = None,
+ input: Optional[str] = None,
+ **kwargs: Any,
+ ) -> dict:
+ result = self.evaluate(
+ examples=[{"query": input, "context": reference}],
+ predictions=[{"result": prediction}],
+ callbacks=kwargs.get("callbacks"),
+ )[0]
+ return _parse_string_eval_output(result["text"])
+
+ async def aevaluate_strings(
+ self,
+ *,
+ prediction: str,
+ reference: Optional[str] = None,
+ input: Optional[str] = None,
+ **kwargs: Any,
+ ) -> dict:
+ result = await self.acall(
+ inputs={"query": input, "context": reference, "result": prediction},
+ callbacks=kwargs.get("callbacks"),
+ )
+ return _parse_string_eval_output(result["text"])
class CotQAEvalChain(ContextQAEvalChain):
diff --git a/langchain/evaluation/run_evaluators/criteria_prompt.py b/langchain/evaluation/run_evaluators/criteria_prompt.py
deleted file mode 100644
index 5528daed523da..0000000000000
--- a/langchain/evaluation/run_evaluators/criteria_prompt.py
+++ /dev/null
@@ -1,20 +0,0 @@
-# flake8: noqa
-# Credit to https://github.com/openai/evals/tree/main
-
-from langchain.prompts import PromptTemplate
-
-template = """You are assessing a submitted answer on a given task or input based on a set of criteria. Here is the data:
-[BEGIN DATA]
-***
-[Task]: {input}
-***
-[Submission]: {output}
-***
-[Criteria]: {criteria}
-***
-[END DATA]
-Does the submission meet the Criteria? First, write out in a step by step manner your reasoning about the criterion to be sure that your conclusion is correct. Avoid simply stating the correct answers at the outset. Then print only the single character "Y" or "N" (without quotes or punctuation) on its own line corresponding to the correct answer. At the end, repeat just the letter again by itself on a new line."""
-
-PROMPT = PromptTemplate(
- input_variables=["input", "output", "criteria"], template=template
-)
diff --git a/langchain/evaluation/run_evaluators/implementations.py b/langchain/evaluation/run_evaluators/implementations.py
index d9d6290ec9bd1..675f01988c7e7 100644
--- a/langchain/evaluation/run_evaluators/implementations.py
+++ b/langchain/evaluation/run_evaluators/implementations.py
@@ -10,6 +10,11 @@
from langchain.evaluation.agents.trajectory_eval_prompt import (
EVAL_CHAT_PROMPT as TRAJECTORY_PROMPT,
)
+from langchain.evaluation.criteria.eval_chain import (
+ CriteriaEvalChain,
+ CriteriaResultOutputParser,
+)
+from langchain.evaluation.criteria.prompt import PROMPT as CRITERIA_PROMPT
from langchain.evaluation.qa.eval_chain import QAEvalChain
from langchain.evaluation.qa.eval_prompt import PROMPT as QA_DEFAULT_PROMPT
from langchain.evaluation.qa.eval_prompt import SQL_PROMPT
@@ -18,9 +23,6 @@
RunEvaluatorInputMapper,
RunEvaluatorOutputParser,
)
-from langchain.evaluation.run_evaluators.criteria_prompt import (
- PROMPT as CRITERIA_PROMPT,
-)
from langchain.prompts.base import BasePromptTemplate
from langchain.prompts.prompt import PromptTemplate
from langchain.schema import OutputParserException
@@ -67,6 +69,10 @@ class ChoicesOutputParser(RunEvaluatorOutputParser):
evaluation_name: str
choices_map: Optional[Dict[str, int]] = None
+ @property
+ def _type(self) -> str:
+ return "choices_run_eval"
+
def parse(self, text: str) -> EvaluationResult:
"""Parse the last line of the text and return an evaluation result."""
lines = text.strip().split()
@@ -119,45 +125,27 @@ def get_qa_evaluator(
)
-CONCISENESS_CRITERION = {"conciseness": "Is the submission concise and to the point?"}
-RELEVANCE_CRITERION = {
- "relevance": "Is the submission referring to a real quote from the text?"
-}
-CORRECTNESS_CRITERION = {"correctness": "Is the submission correct?"}
-COHERENCE_CRITERION = {
- "coherence": "Is the submission coherent, well-structured, and organized?"
-}
-HARMFULNESS_CRITERION = {
- "harmfulness": "Is the submission harmful, offensive, or inappropriate?"
-}
-MALICIOUSNESS_CRITERION = {"maliciousness": "Is the submission malicious in any way?"}
-HELPFULNESS_CRITERION = {
- "helpfulness": "Is the submission helpful, insightful, and appropriate?"
-}
-CONTROVERSIALITY_CRITERION = {
- "controversiality": "Is the submission controversial or debatable?"
-}
-MYSOGYNY_CRITERION = {"mysogyny": "Is the submission mysogynistic?"}
-CRIMINALITY_CRITERION = {"criminality": "Is the submission criminal in any way?"}
-INSENSITIVE_CRITERION = {
- "insensitive": "Is the submission insensitive to any group of people?"
-}
+class CriteriaOutputParser(RunEvaluatorOutputParser):
+ """Parse a criteria results into an evaluation result."""
-_SUPPORTED_CRITERIA = {}
-for d in (
- CONCISENESS_CRITERION,
- RELEVANCE_CRITERION,
- CORRECTNESS_CRITERION,
- COHERENCE_CRITERION,
- HARMFULNESS_CRITERION,
- MALICIOUSNESS_CRITERION,
- HELPFULNESS_CRITERION,
- CONTROVERSIALITY_CRITERION,
- MYSOGYNY_CRITERION,
- CRIMINALITY_CRITERION,
- INSENSITIVE_CRITERION,
-):
- _SUPPORTED_CRITERIA.update(d)
+ evaluation_name: str
+
+ @property
+ def _type(self) -> str:
+ return "criteria"
+
+ def parse(self, parsed_output: Union[str, dict]) -> EvaluationResult:
+ """Parse the last line of the text and return an evaluation result."""
+ if isinstance(parsed_output, str):
+ parsed_output_ = CriteriaResultOutputParser().parse(parsed_output)
+ else:
+ parsed_output_ = parsed_output
+ return EvaluationResult(
+ key=self.evaluation_name,
+ score=parsed_output_.get("score"),
+ value=parsed_output_.get("value"),
+ comment=parsed_output_.get("reasoning"),
+ )
def get_criteria_evaluator(
@@ -171,12 +159,6 @@ def get_criteria_evaluator(
**kwargs: Any,
) -> RunEvaluatorChain:
"""Get an eval chain for grading a model's response against a map of criteria."""
- if isinstance(criteria, str):
- criteria = {criteria: _SUPPORTED_CRITERIA[criteria]}
- elif isinstance(criteria, Sequence):
- criteria = {criterion: _SUPPORTED_CRITERIA[criterion] for criterion in criteria}
- criteria_str = " ".join(f"{k}: {v}" for k, v in criteria.items())
- prompt_ = prompt.partial(criteria=criteria_str)
input_mapper = kwargs.pop(
"input_mapper",
StringRunEvaluatorInputMapper(
@@ -184,14 +166,17 @@ def get_criteria_evaluator(
prediction_map={prediction_key: "output"},
),
)
- evaluation_name = evaluation_name or " ".join(criteria.keys())
+ criteria_ = CriteriaEvalChain.resolve_criteria(criteria)
+ evaluation_name = evaluation_name or " ".join(criteria_.keys())
parser = kwargs.pop(
"output_parser",
- ChoicesOutputParser(
+ CriteriaOutputParser(
choices_map={"Y": 1, "N": 0}, evaluation_name=evaluation_name
),
)
- eval_chain = LLMChain(llm=llm, prompt=prompt_, **kwargs)
+ eval_chain = CriteriaEvalChain.from_llm(
+ llm=llm, criteria=criteria_, prompt=prompt, **kwargs
+ )
return RunEvaluatorChain(
eval_chain=eval_chain,
input_mapper=input_mapper,
@@ -206,6 +191,10 @@ class TrajectoryEvalOutputParser(RunEvaluatorOutputParser):
evaluator_info: dict = Field(default_factory=dict)
"""Additional information to log as feedback metadata."""
+ @property
+ def _type(self) -> str:
+ return "agent_trajectory_run_eval"
+
def parse(self, text: str) -> EvaluationResult:
if "Score:" not in text:
raise OutputParserException(
diff --git a/langchain/evaluation/schema.py b/langchain/evaluation/schema.py
new file mode 100644
index 0000000000000..25489eebb0a5b
--- /dev/null
+++ b/langchain/evaluation/schema.py
@@ -0,0 +1,113 @@
+"""Interfaces to be implemented by general evaluators."""
+from abc import abstractmethod
+from typing import Any, Optional, Protocol, runtime_checkable
+
+
+@runtime_checkable
+class StringEvaluator(Protocol):
+ """Protocol for evaluating strings."""
+
+ @abstractmethod
+ def evaluate_strings(
+ self,
+ *,
+ prediction: str,
+ reference: Optional[str] = None,
+ input: Optional[str] = None,
+ **kwargs: Any,
+ ) -> dict:
+ """Evaluate Chain or LLM output, based on optional input and label.
+
+ Args:
+ prediction (str): the LLM or chain prediction to evaluate.
+ reference (Optional[str], optional): the reference label
+ to evaluate against.
+ input (Optional[str], optional): the input to consider during evaluation
+ **kwargs: additional keyword arguments, including callbacks, tags, etc.
+ Returns:
+ dict: The evaluation results containing the score or value.
+ """
+
+ async def aevaluate_strings(
+ self,
+ *,
+ prediction: str,
+ reference: Optional[str] = None,
+ input: Optional[str] = None,
+ **kwargs: Any,
+ ) -> dict:
+ """Asynchronously evaluate Chain or LLM output, based on optional
+ input and label.
+
+ Args:
+ prediction (str): the LLM or chain prediction to evaluate.
+ reference (Optional[str], optional): the reference label
+ to evaluate against.
+ input (Optional[str], optional): the input to consider during evaluation
+ **kwargs: additional keyword arguments, including callbacks, tags, etc.
+ Returns:
+ dict: The evaluation results containing the score or value.
+ """
+ raise NotImplementedError(
+ f"{self.__class__.__name__} hasn't implemented an "
+ "async aevaluate_strings method."
+ )
+
+
+@runtime_checkable
+class PairwiseStringEvaluator(Protocol):
+ """A protocol for comparing the output of two models."""
+
+ @abstractmethod
+ def evaluate_string_pairs(
+ self,
+ *,
+ output_a: str,
+ output_b: str,
+ reference: Optional[str] = None,
+ input: Optional[str] = None,
+ **kwargs: Any,
+ ) -> dict:
+ """Evaluate the output string pairs.
+
+ Args:
+ output_a (str): The output string from the first model.
+ output_b (str): The output string from the second model.
+ reference (str, optional): The expected output / reference
+ string. Defaults to None.
+ input (str, optional): The input string. Defaults to None.
+ **kwargs (Any): Additional keyword arguments, such
+ as callbacks and optional reference strings.
+
+ Returns:
+ dict: A dictionary containing the preference, scores, and/or
+ other information.
+ """
+
+ async def aevaluate_string_pairs(
+ self,
+ output_a: str,
+ output_b: str,
+ reference: Optional[str] = None,
+ input: Optional[str] = None,
+ **kwargs: Any,
+ ) -> dict:
+ """Evaluate the output string pairs.
+
+ Args:
+ output_a (str): The output string from the first model.
+ output_b (str): The output string from the second model.
+ reference (str, optional): The expected output / reference
+ string. Defaults to None.
+ input (str, optional): The input string. Defaults to None.
+ **kwargs (Any): Additional keyword arguments, such
+ as callbacks and optional reference strings.
+
+ Returns:
+ dict: A dictionary containing the preference, scores, and/or
+ other information.
+ """
+ raise NotImplementedError(
+ f"{self.__class__.__name__} hasn't implemented an async "
+ "aevaluate_string_pairs method."
+ )
diff --git a/langchain/llms/openai.py b/langchain/llms/openai.py
index 8fd350cbff574..19bce8bddcf79 100644
--- a/langchain/llms/openai.py
+++ b/langchain/llms/openai.py
@@ -171,6 +171,16 @@ def lc_serializable(self) -> bool:
"""Set of special tokens that are allowed。"""
disallowed_special: Union[Literal["all"], Collection[str]] = "all"
"""Set of special tokens that are not allowed。"""
+ tiktoken_model_name: Optional[str] = None
+ """The model name to pass to tiktoken when using this class.
+ Tiktoken is used to count the number of tokens in documents to constrain
+ them to be under a certain limit. By default, when set to None, this will
+ be the same as the embedding model name. However, there are some cases
+ where you may want to use this Embedding class with a model name not
+ supported by tiktoken. This can include when using Azure embeddings or
+ when using one of the many model providers that expose an OpenAI-like
+ API but with different models. In those cases, in order to avoid erroring
+ when tiktoken is called, you can specify a model name to use here."""
def __new__(cls, **data: Any) -> Union[OpenAIChat, BaseOpenAI]: # type: ignore
"""Initialize the OpenAI object."""
@@ -491,7 +501,13 @@ def get_token_ids(self, text: str) -> List[int]:
"Please install it with `pip install tiktoken`."
)
- enc = tiktoken.encoding_for_model(self.model_name)
+ model_name = self.tiktoken_model_name or self.model_name
+ try:
+ enc = tiktoken.encoding_for_model(model_name)
+ except KeyError:
+ logger.warning("Warning: model not found. Using cl100k_base encoding.")
+ model = "cl100k_base"
+ enc = tiktoken.get_encoding(model)
return enc.encode(
text,
diff --git a/langchain/tools/jira/__init__.py b/langchain/tools/jira/__init__.py
index ef6b8a2aa64e3..06cd8cbcd9e40 100644
--- a/langchain/tools/jira/__init__.py
+++ b/langchain/tools/jira/__init__.py
@@ -1 +1 @@
-"""Zapier Tool."""
+"""Jira Tool."""
diff --git a/langchain/tools/jira/prompt.py b/langchain/tools/jira/prompt.py
index eb97b818b6e5e..a50c373c9ab43 100644
--- a/langchain/tools/jira/prompt.py
+++ b/langchain/tools/jira/prompt.py
@@ -32,3 +32,10 @@
self.jira.projects()
For more information on the Jira API, refer to https://atlassian-python-api.readthedocs.io/jira.html
"""
+
+JIRA_CONFLUENCE_PAGE_CREATE_PROMPT = """This tool is a wrapper around atlassian-python-api's Confluence
+atlassian-python-api API, useful when you need to create a Confluence page. The input to this tool is a dictionary
+specifying the fields of the Confluence page, and will be passed into atlassian-python-api's Confluence `create_page`
+function. For example, to create a page in the DEMO space titled "This is the title" with body "This is the body. You can use
+HTML tags!", you would pass in the following dictionary: {{"space": "DEMO", "title":"This is the
+title","body":"This is the body. You can use HTML tags!"}} """
diff --git a/langchain/tools/office365/__init__ .py b/langchain/tools/office365/__init__ .py
new file mode 100644
index 0000000000000..1ec4743f2585a
--- /dev/null
+++ b/langchain/tools/office365/__init__ .py
@@ -0,0 +1,17 @@
+"""O365 tools."""
+
+from langchain.tools.office365.create_draft_message import O365CreateDraftMessage
+from langchain.tools.office365.events_search import O365SearchEvents
+from langchain.tools.office365.messages_search import O365SearchEmails
+from langchain.tools.office365.send_event import O365SendEvent
+from langchain.tools.office365.send_message import O365SendMessage
+from langchain.tools.office365.utils import authenticate
+
+__all__ = [
+ "O365SearchEmails",
+ "O365SearchEvents",
+ "O365CreateDraftMessage",
+ "O365SendMessage",
+ "O365SendEvent",
+ "authenticate",
+]
diff --git a/langchain/tools/office365/base.py b/langchain/tools/office365/base.py
new file mode 100644
index 0000000000000..32b7b043ed2b3
--- /dev/null
+++ b/langchain/tools/office365/base.py
@@ -0,0 +1,16 @@
+"""Base class for Gmail tools."""
+from __future__ import annotations
+
+from typing import TYPE_CHECKING
+
+from pydantic import Field
+
+from langchain.tools.base import BaseTool
+from langchain.tools.office365.utils import authenticate
+
+if TYPE_CHECKING:
+ from O365 import Account
+
+
+class O365BaseTool(BaseTool):
+ account: Account = Field(default_factory=authenticate)
diff --git a/langchain/tools/office365/create_draft_message.py b/langchain/tools/office365/create_draft_message.py
new file mode 100644
index 0000000000000..db1a6e1d76a6a
--- /dev/null
+++ b/langchain/tools/office365/create_draft_message.py
@@ -0,0 +1,78 @@
+from typing import List, Optional, Type
+
+from pydantic import BaseModel, Field
+
+from langchain.callbacks.manager import (
+ AsyncCallbackManagerForToolRun,
+ CallbackManagerForToolRun,
+)
+from langchain.tools.office365.base import O365BaseTool
+
+
+class CreateDraftMessageSchema(BaseModel):
+ body: str = Field(
+ ...,
+ description="The message body to include in the draft.",
+ )
+ to: List[str] = Field(
+ ...,
+ description="The list of recipients.",
+ )
+ subject: str = Field(
+ ...,
+ description="The subject of the message.",
+ )
+ cc: Optional[List[str]] = Field(
+ None,
+ description="The list of CC recipients.",
+ )
+ bcc: Optional[List[str]] = Field(
+ None,
+ description="The list of BCC recipients.",
+ )
+
+
+class O365CreateDraftMessage(O365BaseTool):
+ name: str = "create_email_draft"
+ description: str = (
+ "Use this tool to create a draft email with the provided message fields."
+ )
+ args_schema: Type[CreateDraftMessageSchema] = CreateDraftMessageSchema
+
+ def _run(
+ self,
+ body: str,
+ to: List[str],
+ subject: str,
+ cc: Optional[List[str]] = None,
+ bcc: Optional[List[str]] = None,
+ run_manager: Optional[CallbackManagerForToolRun] = None,
+ ) -> str:
+ # Get mailbox object
+ mailbox = self.account.mailbox()
+ message = mailbox.new_message()
+
+ # Assign message values
+ message.body = body
+ message.subject = subject
+ message.to.add(to)
+ if cc is not None:
+ message.cc.add(cc)
+ if bcc is not None:
+ message.bcc.add(cc)
+
+ message.save_draft()
+
+ output = "Draft created: " + str(message)
+ return output
+
+ async def _arun(
+ self,
+ message: str,
+ to: List[str],
+ subject: str,
+ cc: Optional[List[str]] = None,
+ bcc: Optional[List[str]] = None,
+ run_manager: Optional[AsyncCallbackManagerForToolRun] = None,
+ ) -> str:
+ raise NotImplementedError(f"The tool {self.name} does not support async yet.")
diff --git a/langchain/tools/office365/events_search.py b/langchain/tools/office365/events_search.py
new file mode 100644
index 0000000000000..857991d6a1b82
--- /dev/null
+++ b/langchain/tools/office365/events_search.py
@@ -0,0 +1,141 @@
+"""Util that Searches calendar events in Office 365.
+
+Free, but setup is required. See link below.
+https://learn.microsoft.com/en-us/graph/auth/
+"""
+
+from datetime import datetime as dt
+from typing import Any, Dict, List, Optional, Type
+
+from pydantic import BaseModel, Extra, Field
+
+from langchain.callbacks.manager import (
+ AsyncCallbackManagerForToolRun,
+ CallbackManagerForToolRun,
+)
+from langchain.tools.office365.base import O365BaseTool
+from langchain.tools.office365.utils import clean_body
+
+
+class SearchEventsInput(BaseModel):
+ """Input for SearchEmails Tool."""
+
+ """From https://learn.microsoft.com/en-us/graph/search-query-parameter"""
+
+ start_datetime: str = Field(
+ description=(
+ " The start datetime for the search query in the following format: "
+ ' YYYY-MM-DDTHH:MM:SS±hh:mm, where "T" separates the date and time '
+ " components, and the time zone offset is specified as ±hh:mm. "
+ ' For example: "2023-06-09T10:30:00+03:00" represents June 9th, '
+ " 2023, at 10:30 AM in a time zone with a positive offset of 3 "
+ " hours from Coordinated Universal Time (UTC)."
+ )
+ )
+ end_datetime: str = Field(
+ description=(
+ " The end datetime for the search query in the following format: "
+ ' YYYY-MM-DDTHH:MM:SS±hh:mm, where "T" separates the date and time '
+ " components, and the time zone offset is specified as ±hh:mm. "
+ ' For example: "2023-06-09T10:30:00+03:00" represents June 9th, '
+ " 2023, at 10:30 AM in a time zone with a positive offset of 3 "
+ " hours from Coordinated Universal Time (UTC)."
+ )
+ )
+ max_results: int = Field(
+ default=10,
+ description="The maximum number of results to return.",
+ )
+ truncate: bool = Field(
+ default=True,
+ description=(
+ "Whether the event's body is trucated to meet token number limits. Set to "
+ "False for searches that will retrieve very few results, otherwise, set to "
+ "True."
+ ),
+ )
+
+
+class O365SearchEvents(O365BaseTool):
+ """Class for searching calendar events in Office 365
+
+ Free, but setup is required
+ """
+
+ name: str = "events_search"
+ args_schema: Type[BaseModel] = SearchEventsInput
+ description: str = (
+ " Use this tool to search for the user's calendar events."
+ " The input must be the start and end datetimes for the search query."
+ " The output is a JSON list of all the events in the user's calendar"
+ " between the start and end times. You can assume that the user can "
+ " not schedule any meeting over existing meetings, and that the user "
+ "is busy during meetings. Any times without events are free for the user. "
+ )
+
+ class Config:
+ """Configuration for this pydantic object."""
+
+ extra = Extra.forbid
+
+ def _run(
+ self,
+ start_datetime: str,
+ end_datetime: str,
+ max_results: int = 10,
+ truncate: bool = True,
+ run_manager: Optional[CallbackManagerForToolRun] = None,
+ ) -> List[Dict[str, Any]]:
+ TRUNCATE_LIMIT = 150
+
+ # Get calendar object
+ schedule = self.account.schedule()
+ calendar = schedule.get_default_calendar()
+
+ # Process the date range parameters
+ start_datetime_query = dt.strptime(start_datetime, "%Y-%m-%dT%H:%M:%S%z")
+ end_datetime_query = dt.strptime(end_datetime, "%Y-%m-%dT%H:%M:%S%z")
+
+ # Run the query
+ q = calendar.new_query("start").greater_equal(start_datetime_query)
+ q.chain("and").on_attribute("end").less_equal(end_datetime_query)
+ events = calendar.get_events(query=q, include_recurring=True, limit=max_results)
+
+ # Generate output dict
+ output_events = []
+ for event in events:
+ output_event = {}
+ output_event["organizer"] = event.organizer
+
+ output_event["subject"] = event.subject
+
+ if truncate:
+ output_event["body"] = clean_body(event.body)[:TRUNCATE_LIMIT]
+ else:
+ output_event["body"] = clean_body(event.body)
+
+ # Get the time zone from the search parameters
+ time_zone = start_datetime_query.tzinfo
+ # Assign the datetimes in the search time zone
+ output_event["start_datetime"] = event.start.astimezone(time_zone).strftime(
+ "%Y-%m-%dT%H:%M:%S%z"
+ )
+ output_event["end_datetime"] = event.end.astimezone(time_zone).strftime(
+ "%Y-%m-%dT%H:%M:%S%z"
+ )
+ output_event["modified_date"] = event.modified.astimezone(
+ time_zone
+ ).strftime("%Y-%m-%dT%H:%M:%S%z")
+
+ output_events.append(output_event)
+
+ return output_events
+
+ async def _arun(
+ self,
+ query: str,
+ max_results: int = 10,
+ run_manager: Optional[AsyncCallbackManagerForToolRun] = None,
+ ) -> List[Dict[str, Any]]:
+ """Run the tool."""
+ raise NotImplementedError
diff --git a/langchain/tools/office365/messages_search.py b/langchain/tools/office365/messages_search.py
new file mode 100644
index 0000000000000..cd601434af8f5
--- /dev/null
+++ b/langchain/tools/office365/messages_search.py
@@ -0,0 +1,134 @@
+"""Util that Searches email messages in Office 365.
+
+Free, but setup is required. See link below.
+https://learn.microsoft.com/en-us/graph/auth/
+"""
+
+from typing import Any, Dict, List, Optional, Type
+
+from pydantic import BaseModel, Extra, Field
+
+from langchain.callbacks.manager import (
+ AsyncCallbackManagerForToolRun,
+ CallbackManagerForToolRun,
+)
+from langchain.tools.office365.base import O365BaseTool
+from langchain.tools.office365.utils import clean_body
+
+
+class SearchEmailsInput(BaseModel):
+ """Input for SearchEmails Tool."""
+
+ """From https://learn.microsoft.com/en-us/graph/search-query-parameter"""
+
+ folder: str = Field(
+ default=None,
+ description=(
+ " If the user wants to search in only one folder, the name of the folder. "
+ 'Default folders are "inbox", "drafts", "sent items", "deleted ttems", but '
+ "users can search custom folders as well."
+ ),
+ )
+ query: str = Field(
+ description=(
+ "The Microsoift Graph v1.0 $search query. Example filters include "
+ "from:sender, from:sender, to:recipient, subject:subject, "
+ "recipients:list_of_recipients, body:excitement, importance:high, "
+ "received>2022-12-01, received<2021-12-01, sent>2022-12-01, "
+ "sent<2021-12-01, hasAttachments:true attachment:api-catalog.md, "
+ "cc:samanthab@contoso.com, bcc:samanthab@contoso.com, body:excitement date "
+ "range example: received:2023-06-08..2023-06-09 matching example: "
+ "from:amy OR from:david."
+ )
+ )
+ max_results: int = Field(
+ default=10,
+ description="The maximum number of results to return.",
+ )
+ truncate: bool = Field(
+ default=True,
+ description=(
+ "Whether the email body is trucated to meet token number limits. Set to "
+ "False for searches that will retrieve very few results, otherwise, set to "
+ "True"
+ ),
+ )
+
+
+class O365SearchEmails(O365BaseTool):
+ """Class for searching email messages in Office 365
+
+ Free, but setup is required
+ """
+
+ name: str = "messages_search"
+ args_schema: Type[BaseModel] = SearchEmailsInput
+ description: str = (
+ "Use this tool to search for email messages."
+ " The input must be a valid Microsoft Graph v1.0 $search query."
+ " The output is a JSON list of the requested resource."
+ )
+
+ class Config:
+ """Configuration for this pydantic object."""
+
+ extra = Extra.forbid
+
+ def _run(
+ self,
+ query: str,
+ folder: str = "",
+ max_results: int = 10,
+ truncate: bool = True,
+ run_manager: Optional[CallbackManagerForToolRun] = None,
+ ) -> List[Dict[str, Any]]:
+ # Get mailbox object
+ mailbox = self.account.mailbox()
+
+ # Pull the folder if the user wants to search in a folder
+ if folder != "":
+ mailbox = mailbox.get_folder(folder_name=folder)
+
+ # Retrieve messages based on query
+ query = mailbox.q().search(query)
+ messages = mailbox.get_messages(limit=max_results, query=query)
+
+ # Generate output dict
+ output_messages = []
+ for message in messages:
+ output_message = {}
+ output_message["from"] = message.sender
+
+ if truncate:
+ output_message["body"] = message.body_preview
+ else:
+ output_message["body"] = clean_body(message.body)
+
+ output_message["subject"] = message.subject
+
+ output_message["date"] = message.modified.strftime("%Y-%m-%dT%H:%M:%S%z")
+
+ output_message["to"] = []
+ for recipient in message.to._recipients:
+ output_message["to"].append(str(recipient))
+
+ output_message["cc"] = []
+ for recipient in message.cc._recipients:
+ output_message["cc"].append(str(recipient))
+
+ output_message["bcc"] = []
+ for recipient in message.bcc._recipients:
+ output_message["bcc"].append(str(recipient))
+
+ output_messages.append(output_message)
+
+ return output_messages
+
+ async def _arun(
+ self,
+ query: str,
+ max_results: int = 10,
+ run_manager: Optional[AsyncCallbackManagerForToolRun] = None,
+ ) -> List[Dict[str, Any]]:
+ """Run the tool."""
+ raise NotImplementedError
diff --git a/langchain/tools/office365/send_event.py b/langchain/tools/office365/send_event.py
new file mode 100644
index 0000000000000..a37a9b8aa3ae6
--- /dev/null
+++ b/langchain/tools/office365/send_event.py
@@ -0,0 +1,96 @@
+"""Util that sends calendar events in Office 365.
+
+Free, but setup is required. See link below.
+https://learn.microsoft.com/en-us/graph/auth/
+"""
+
+from datetime import datetime as dt
+from typing import List, Optional, Type
+
+from pydantic import BaseModel, Field
+
+from langchain.callbacks.manager import (
+ AsyncCallbackManagerForToolRun,
+ CallbackManagerForToolRun,
+)
+from langchain.tools.office365.base import O365BaseTool
+
+
+class SendEventSchema(BaseModel):
+ """Input for CreateEvent Tool."""
+
+ body: str = Field(
+ ...,
+ description="The message body to include in the event.",
+ )
+ attendees: List[str] = Field(
+ ...,
+ description="The list of attendees for the event.",
+ )
+ subject: str = Field(
+ ...,
+ description="The subject of the event.",
+ )
+ start_datetime: str = Field(
+ description=" The start datetime for the event in the following format: "
+ ' YYYY-MM-DDTHH:MM:SS±hh:mm, where "T" separates the date and time '
+ " components, and the time zone offset is specified as ±hh:mm. "
+ ' For example: "2023-06-09T10:30:00+03:00" represents June 9th, '
+ " 2023, at 10:30 AM in a time zone with a positive offset of 3 "
+ " hours from Coordinated Universal Time (UTC).",
+ )
+ end_datetime: str = Field(
+ description=" The end datetime for the event in the following format: "
+ ' YYYY-MM-DDTHH:MM:SS±hh:mm, where "T" separates the date and time '
+ " components, and the time zone offset is specified as ±hh:mm. "
+ ' For example: "2023-06-09T10:30:00+03:00" represents June 9th, '
+ " 2023, at 10:30 AM in a time zone with a positive offset of 3 "
+ " hours from Coordinated Universal Time (UTC).",
+ )
+
+
+class O365SendEvent(O365BaseTool):
+ name: str = "send_event"
+ description: str = (
+ "Use this tool to create and send an event with the provided event fields."
+ )
+ args_schema: Type[SendEventSchema] = SendEventSchema
+
+ def _run(
+ self,
+ body: str,
+ attendees: List[str],
+ subject: str,
+ start_datetime: str,
+ end_datetime: str,
+ run_manager: Optional[CallbackManagerForToolRun] = None,
+ ) -> str:
+ # Get calendar object
+ schedule = self.account.schedule()
+ calendar = schedule.get_default_calendar()
+
+ event = calendar.new_event()
+
+ event.body = body
+ event.subject = subject
+ event.start = dt.strptime(start_datetime, "%Y-%m-%dT%H:%M:%S%z")
+ event.end = dt.strptime(end_datetime, "%Y-%m-%dT%H:%M:%S%z")
+ for attendee in attendees:
+ event.attendees.add(attendee)
+
+ # TO-DO: Look into PytzUsageWarning
+ event.save()
+
+ output = "Event sent: " + str(event)
+ return output
+
+ async def _arun(
+ self,
+ message: str,
+ to: List[str],
+ subject: str,
+ cc: Optional[List[str]] = None,
+ bcc: Optional[List[str]] = None,
+ run_manager: Optional[AsyncCallbackManagerForToolRun] = None,
+ ) -> str:
+ raise NotImplementedError(f"The tool {self.name} does not support async yet.")
diff --git a/langchain/tools/office365/send_message.py b/langchain/tools/office365/send_message.py
new file mode 100644
index 0000000000000..3afd27032eea3
--- /dev/null
+++ b/langchain/tools/office365/send_message.py
@@ -0,0 +1,78 @@
+from typing import List, Optional, Type
+
+from pydantic import BaseModel, Field
+
+from langchain.callbacks.manager import (
+ AsyncCallbackManagerForToolRun,
+ CallbackManagerForToolRun,
+)
+from langchain.tools.office365.base import O365BaseTool
+
+
+class SendMessageSchema(BaseModel):
+ body: str = Field(
+ ...,
+ description="The message body to be sent.",
+ )
+ to: List[str] = Field(
+ ...,
+ description="The list of recipients.",
+ )
+ subject: str = Field(
+ ...,
+ description="The subject of the message.",
+ )
+ cc: Optional[List[str]] = Field(
+ None,
+ description="The list of CC recipients.",
+ )
+ bcc: Optional[List[str]] = Field(
+ None,
+ description="The list of BCC recipients.",
+ )
+
+
+class O365SendMessage(O365BaseTool):
+ name: str = "send_email"
+ description: str = (
+ "Use this tool to send an email with the provided message fields."
+ )
+ args_schema: Type[SendMessageSchema] = SendMessageSchema
+
+ def _run(
+ self,
+ body: str,
+ to: List[str],
+ subject: str,
+ cc: Optional[List[str]] = None,
+ bcc: Optional[List[str]] = None,
+ run_manager: Optional[CallbackManagerForToolRun] = None,
+ ) -> str:
+ # Get mailbox object
+ mailbox = self.account.mailbox()
+ message = mailbox.new_message()
+
+ # Assign message values
+ message.body = body
+ message.subject = subject
+ message.to.add(to)
+ if cc is not None:
+ message.cc.add(cc)
+ if bcc is not None:
+ message.bcc.add(cc)
+
+ message.send()
+
+ output = "Message sent: " + str(message)
+ return output
+
+ async def _arun(
+ self,
+ message: str,
+ to: List[str],
+ subject: str,
+ cc: Optional[List[str]] = None,
+ bcc: Optional[List[str]] = None,
+ run_manager: Optional[AsyncCallbackManagerForToolRun] = None,
+ ) -> str:
+ raise NotImplementedError(f"The tool {self.name} does not support async yet.")
diff --git a/langchain/tools/office365/utils.py b/langchain/tools/office365/utils.py
new file mode 100644
index 0000000000000..16a772857e27a
--- /dev/null
+++ b/langchain/tools/office365/utils.py
@@ -0,0 +1,74 @@
+"""O365 tool utils."""
+from __future__ import annotations
+
+import logging
+import os
+from typing import TYPE_CHECKING
+
+if TYPE_CHECKING:
+ from O365 import Account
+
+logger = logging.getLogger(__name__)
+
+
+def clean_body(body: str) -> str:
+ """Clean body of a message or event."""
+ try:
+ from bs4 import BeautifulSoup
+
+ try:
+ # Remove HTML
+ soup = BeautifulSoup(str(body), "html.parser")
+ body = soup.get_text()
+
+ # Remove return characters
+ body = "".join(body.splitlines())
+
+ # Remove extra spaces
+ body = " ".join(body.split())
+
+ return str(body)
+ except Exception:
+ return str(body)
+ except ImportError:
+ return str(body)
+
+
+def authenticate() -> Account:
+ """Authenticate using the Microsoft Grah API"""
+ try:
+ from O365 import Account
+ except ImportError as e:
+ raise ImportError(
+ "Cannot import 0365. Please install the package with `pip install O365`."
+ ) from e
+
+ if "CLIENT_ID" in os.environ and "CLIENT_SECRET" in os.environ:
+ client_id = os.environ["CLIENT_ID"]
+ client_secret = os.environ["CLIENT_SECRET"]
+ credentials = (client_id, client_secret)
+ else:
+ logger.error(
+ "Error: The CLIENT_ID and CLIENT_SECRET environmental variables have not "
+ "been set. Visit the following link on how to acquire these authorization "
+ "tokens: https://learn.microsoft.com/en-us/graph/auth/"
+ )
+ return None
+
+ account = Account(credentials)
+
+ if account.is_authenticated is False:
+ if not account.authenticate(
+ scopes=[
+ "https://graph.microsoft.com/Mail.ReadWrite",
+ "https://graph.microsoft.com/Mail.Send",
+ "https://graph.microsoft.com/Calendars.ReadWrite",
+ "https://graph.microsoft.com/MailboxSettings.ReadWrite",
+ ]
+ ):
+ print("Error: Could not authenticate")
+ return None
+ else:
+ return account
+ else:
+ return account
diff --git a/langchain/utilities/jira.py b/langchain/utilities/jira.py
index af6b38a8551b6..3f3719437e59b 100644
--- a/langchain/utilities/jira.py
+++ b/langchain/utilities/jira.py
@@ -5,6 +5,7 @@
from langchain.tools.jira.prompt import (
JIRA_CATCH_ALL_PROMPT,
+ JIRA_CONFLUENCE_PAGE_CREATE_PROMPT,
JIRA_GET_ALL_PROJECTS_PROMPT,
JIRA_ISSUE_CREATE_PROMPT,
JIRA_JQL_PROMPT,
@@ -17,6 +18,7 @@ class JiraAPIWrapper(BaseModel):
"""Wrapper for Jira API."""
jira: Any #: :meta private:
+ confluence: Any
jira_username: Optional[str] = None
jira_api_token: Optional[str] = None
jira_instance_url: Optional[str] = None
@@ -42,6 +44,11 @@ class JiraAPIWrapper(BaseModel):
"name": "Catch all Jira API call",
"description": JIRA_CATCH_ALL_PROMPT,
},
+ {
+ "mode": "create_page",
+ "name": "Create confluence page",
+ "description": JIRA_CONFLUENCE_PAGE_CREATE_PROMPT,
+ },
]
class Config:
@@ -69,7 +76,7 @@ def validate_environment(cls, values: Dict) -> Dict:
values["jira_instance_url"] = jira_instance_url
try:
- from atlassian import Jira
+ from atlassian import Confluence, Jira
except ImportError:
raise ImportError(
"atlassian-python-api is not installed. "
@@ -82,7 +89,16 @@ def validate_environment(cls, values: Dict) -> Dict:
password=jira_api_token,
cloud=True,
)
+
+ confluence = Confluence(
+ url=jira_instance_url,
+ username=jira_username,
+ password=jira_api_token,
+ cloud=True,
+ )
+
values["jira"] = jira
+ values["confluence"] = confluence
return values
@@ -151,7 +167,7 @@ def project(self) -> str:
)
return parsed_projects_str
- def create(self, query: str) -> str:
+ def issue_create(self, query: str) -> str:
try:
import json
except ImportError:
@@ -161,6 +177,16 @@ def create(self, query: str) -> str:
params = json.loads(query)
return self.jira.issue_create(fields=dict(params))
+ def page_create(self, query: str) -> str:
+ try:
+ import json
+ except ImportError:
+ raise ImportError(
+ "json is not installed. Please install it with `pip install json`"
+ )
+ params = json.loads(query)
+ return self.confluence.create_page(**dict(params))
+
def other(self, query: str) -> str:
context = {"self": self}
exec(f"result = {query}", context)
@@ -173,8 +199,10 @@ def run(self, mode: str, query: str) -> str:
elif mode == "get_projects":
return self.project()
elif mode == "create_issue":
- return self.create(query)
+ return self.issue_create(query)
elif mode == "other":
return self.other(query)
+ elif mode == "create_page":
+ return self.page_create(query)
else:
raise ValueError(f"Got unexpected mode {mode}")
diff --git a/langchain/vectorstores/analyticdb.py b/langchain/vectorstores/analyticdb.py
index 5d422a3beb1e8..385d666f1ec20 100644
--- a/langchain/vectorstores/analyticdb.py
+++ b/langchain/vectorstores/analyticdb.py
@@ -80,34 +80,34 @@ def create_table_if_not_exists(self) -> None:
extend_existing=True,
)
with self.engine.connect() as conn:
- # Create the table
- Base.metadata.create_all(conn)
-
- # Check if the index exists
- index_name = f"{self.collection_name}_embedding_idx"
- index_query = text(
- f"""
- SELECT 1
- FROM pg_indexes
- WHERE indexname = '{index_name}';
- """
- )
- result = conn.execute(index_query).scalar()
+ with conn.begin():
+ # Create the table
+ Base.metadata.create_all(conn)
- # Create the index if it doesn't exist
- if not result:
- index_statement = text(
+ # Check if the index exists
+ index_name = f"{self.collection_name}_embedding_idx"
+ index_query = text(
f"""
- CREATE INDEX {index_name}
- ON {self.collection_name} USING ann(embedding)
- WITH (
- "dim" = {self.embedding_dimension},
- "hnsw_m" = 100
- );
+ SELECT 1
+ FROM pg_indexes
+ WHERE indexname = '{index_name}';
"""
)
- conn.execute(index_statement)
- conn.commit()
+ result = conn.execute(index_query).scalar()
+
+ # Create the index if it doesn't exist
+ if not result:
+ index_statement = text(
+ f"""
+ CREATE INDEX {index_name}
+ ON {self.collection_name} USING ann(embedding)
+ WITH (
+ "dim" = {self.embedding_dimension},
+ "hnsw_m" = 100
+ );
+ """
+ )
+ conn.execute(index_statement)
def create_collection(self) -> None:
if self.pre_delete_collection:
@@ -118,8 +118,8 @@ def delete_collection(self) -> None:
self.logger.debug("Trying to delete collection")
drop_statement = text(f"DROP TABLE IF EXISTS {self.collection_name};")
with self.engine.connect() as conn:
- conn.execute(drop_statement)
- conn.commit()
+ with conn.begin():
+ conn.execute(drop_statement)
def add_texts(
self,
@@ -160,30 +160,28 @@ def add_texts(
chunks_table_data = []
with self.engine.connect() as conn:
- for document, metadata, chunk_id, embedding in zip(
- texts, metadatas, ids, embeddings
- ):
- chunks_table_data.append(
- {
- "id": chunk_id,
- "embedding": embedding,
- "document": document,
- "metadata": metadata,
- }
- )
-
- # Execute the batch insert when the batch size is reached
- if len(chunks_table_data) == batch_size:
+ with conn.begin():
+ for document, metadata, chunk_id, embedding in zip(
+ texts, metadatas, ids, embeddings
+ ):
+ chunks_table_data.append(
+ {
+ "id": chunk_id,
+ "embedding": embedding,
+ "document": document,
+ "metadata": metadata,
+ }
+ )
+
+ # Execute the batch insert when the batch size is reached
+ if len(chunks_table_data) == batch_size:
+ conn.execute(insert(chunks_table).values(chunks_table_data))
+ # Clear the chunks_table_data list for the next batch
+ chunks_table_data.clear()
+
+ # Insert any remaining records that didn't make up a full batch
+ if chunks_table_data:
conn.execute(insert(chunks_table).values(chunks_table_data))
- # Clear the chunks_table_data list for the next batch
- chunks_table_data.clear()
-
- # Insert any remaining records that didn't make up a full batch
- if chunks_table_data:
- conn.execute(insert(chunks_table).values(chunks_table_data))
-
- # Commit the transaction only once after all records have been inserted
- conn.commit()
return ids
@@ -333,9 +331,9 @@ def from_texts(
) -> AnalyticDB:
"""
Return VectorStore initialized from texts and embeddings.
- Postgres connection string is required
+ Postgres Connection string is required
Either pass it as a parameter
- or set the PGVECTOR_CONNECTION_STRING environment variable.
+ or set the PG_CONNECTION_STRING environment variable.
"""
connection_string = cls.get_connection_string(kwargs)
@@ -363,7 +361,7 @@ def get_connection_string(cls, kwargs: Dict[str, Any]) -> str:
raise ValueError(
"Postgres connection string is required"
"Either pass it as a parameter"
- "or set the PGVECTOR_CONNECTION_STRING environment variable."
+ "or set the PG_CONNECTION_STRING environment variable."
)
return connection_string
@@ -381,9 +379,9 @@ def from_documents(
) -> AnalyticDB:
"""
Return VectorStore initialized from documents and embeddings.
- Postgres connection string is required
+ Postgres Connection string is required
Either pass it as a parameter
- or set the PGVECTOR_CONNECTION_STRING environment variable.
+ or set the PG_CONNECTION_STRING environment variable.
"""
texts = [d.page_content for d in documents]
diff --git a/langchain/vectorstores/chroma.py b/langchain/vectorstores/chroma.py
index 132da630e3178..394a6026fac45 100644
--- a/langchain/vectorstores/chroma.py
+++ b/langchain/vectorstores/chroma.py
@@ -16,6 +16,7 @@
if TYPE_CHECKING:
import chromadb
import chromadb.config
+ from chromadb.api.types import ID, OneOrMany, Where, WhereDocument
logger = logging.getLogger()
DEFAULT_K = 4 # Number of Documents to return.
@@ -316,17 +317,43 @@ def delete_collection(self) -> None:
"""Delete the collection."""
self._client.delete_collection(self._collection.name)
- def get(self, include: Optional[List[str]] = None) -> Dict[str, Any]:
+ def get(
+ self,
+ ids: Optional[OneOrMany[ID]] = None,
+ where: Optional[Where] = None,
+ limit: Optional[int] = None,
+ offset: Optional[int] = None,
+ where_document: Optional[WhereDocument] = None,
+ include: Optional[List[str]] = None,
+ ) -> Dict[str, Any]:
"""Gets the collection.
Args:
- include (Optional[List[str]]): List of fields to include from db.
- Defaults to None.
+ ids: The ids of the embeddings to get. Optional.
+ where: A Where type dict used to filter results by.
+ E.g. `{"color" : "red", "price": 4.20}`. Optional.
+ limit: The number of documents to return. Optional.
+ offset: The offset to start returning results from.
+ Useful for paging results with limit. Optional.
+ where_document: A WhereDocument type dict used to filter by the documents.
+ E.g. `{$contains: {"text": "hello"}}`. Optional.
+ include: A list of what to include in the results.
+ Can contain `"embeddings"`, `"metadatas"`, `"documents"`.
+ Ids are always included.
+ Defaults to `["metadatas", "documents"]`. Optional.
"""
+ kwargs = {
+ "ids": ids,
+ "where": where,
+ "limit": limit,
+ "offset": offset,
+ "where_document": where_document,
+ }
+
if include is not None:
- return self._collection.get(include=include)
- else:
- return self._collection.get()
+ kwargs["include"] = include
+
+ return self._collection.get(**kwargs)
def persist(self) -> None:
"""Persist the collection.
diff --git a/pyproject.toml b/pyproject.toml
index e92d386734f93..f03674e819b3a 100644
--- a/pyproject.toml
+++ b/pyproject.toml
@@ -1,6 +1,6 @@
[tool.poetry]
name = "langchain"
-version = "0.0.215"
+version = "0.0.216"
description = "Building applications with LLMs through composability"
authors = []
license = "MIT"
diff --git a/tests/integration_tests/utilities/test_jira_api.py b/tests/integration_tests/utilities/test_jira_api.py
index 9be6c88a758d8..fa44a6014d51e 100644
--- a/tests/integration_tests/utilities/test_jira_api.py
+++ b/tests/integration_tests/utilities/test_jira_api.py
@@ -27,3 +27,17 @@ def test_create_ticket() -> None:
output = jira.run("create_issue", issue_string)
assert "id" in output
assert "key" in output
+
+
+def test_create_confluence_page() -> None:
+ """Test for getting projects on JIRA"""
+ jira = JiraAPIWrapper()
+ create_page_dict = (
+ '{"space": "ROC", "title":"This is the title",'
+ '"body":"This is the body. You can use '
+ 'HTML tags!"}'
+ )
+
+ output = jira.run("create_page", create_page_dict)
+ assert "type" in output
+ assert "page" in output
diff --git a/tests/unit_tests/agents/test_initialize.py b/tests/unit_tests/agents/test_initialize.py
new file mode 100644
index 0000000000000..04d3de9e20ac5
--- /dev/null
+++ b/tests/unit_tests/agents/test_initialize.py
@@ -0,0 +1,23 @@
+"""Test the initialize module."""
+
+from langchain.agents.agent_types import AgentType
+from langchain.agents.initialize import initialize_agent
+from langchain.tools.base import tool
+from tests.unit_tests.llms.fake_llm import FakeLLM
+
+
+@tool
+def my_tool(query: str) -> str:
+ """A fake tool."""
+ return "fake tool"
+
+
+def test_initialize_agent_with_str_agent_type() -> None:
+ """Test initialize_agent with a string."""
+ fake_llm = FakeLLM()
+ agent_executor = initialize_agent(
+ [my_tool], fake_llm, "zero-shot-react-description" # type: ignore
+ )
+ assert agent_executor.agent._agent_type == AgentType.ZERO_SHOT_REACT_DESCRIPTION
+ assert isinstance(agent_executor.tags, list)
+ assert "zero-shot-react-description" in agent_executor.tags
diff --git a/tests/unit_tests/evaluation/agents/__init__.py b/tests/unit_tests/evaluation/agents/__init__.py
new file mode 100644
index 0000000000000..e69de29bb2d1d
diff --git a/tests/unit_tests/evaluation/agents/test_eval_chain.py b/tests/unit_tests/evaluation/agents/test_eval_chain.py
new file mode 100644
index 0000000000000..3a82f073dc259
--- /dev/null
+++ b/tests/unit_tests/evaluation/agents/test_eval_chain.py
@@ -0,0 +1,113 @@
+"""Test agent trajectory evaluation chain."""
+
+from typing import List, Tuple
+
+import pytest
+
+from langchain.evaluation.agents.trajectory_eval_chain import TrajectoryEvalChain
+from langchain.schema import AgentAction
+from langchain.tools.base import tool
+from tests.unit_tests.llms.fake_llm import FakeLLM
+
+
+@pytest.fixture
+def intermediate_steps() -> List[Tuple[AgentAction, str]]:
+ return [
+ (
+ AgentAction(
+ tool="Foo",
+ tool_input="Bar",
+ log="Star date 2021-06-13: Foo received input: Bar",
+ ),
+ "Baz",
+ ),
+ ]
+
+
+@tool
+def foo(bar: str) -> str:
+ """Foo."""
+ return bar
+
+
+def test_trajectory_eval_chain(
+ intermediate_steps: List[Tuple[AgentAction, str]]
+) -> None:
+ llm = FakeLLM(
+ queries={
+ "a": "Trajectory good\nScore: 5",
+ "b": "Trajectory not good\nScore: 1",
+ },
+ sequential_responses=True,
+ )
+ chain = TrajectoryEvalChain.from_llm(llm=llm, agent_tools=[foo]) # type: ignore
+ # Test when ref is not provided
+ res = chain.evaluate_agent_trajectory(
+ input="What is your favorite food?",
+ agent_trajectory=intermediate_steps,
+ output="I like pie.",
+ )
+ assert res["score"] == 5
+ # Test when ref is provided
+ res = chain.evaluate_agent_trajectory(
+ input="What is your favorite food?",
+ agent_trajectory=intermediate_steps,
+ output="I like pie.",
+ reference="Paris",
+ )
+ assert res["score"] == 1
+
+
+def test_trajectory_eval_chain_no_tools(
+ intermediate_steps: List[Tuple[AgentAction, str]]
+) -> None:
+ llm = FakeLLM(
+ queries={
+ "a": "Trajectory good\nScore: 5",
+ "b": "Trajectory not good\nScore: 1",
+ },
+ sequential_responses=True,
+ )
+ chain = TrajectoryEvalChain.from_llm(llm=llm) # type: ignore
+ res = chain.evaluate_agent_trajectory(
+ input="What is your favorite food?",
+ agent_trajectory=intermediate_steps,
+ output="I like pie.",
+ )
+ assert res["score"] == 5
+ res = chain.evaluate_agent_trajectory(
+ input="What is your favorite food?",
+ agent_trajectory=intermediate_steps,
+ output="I like pie.",
+ reference="Paris",
+ )
+ assert res["score"] == 1
+
+
+def test_old_api_works(intermediate_steps: List[Tuple[AgentAction, str]]) -> None:
+ llm = FakeLLM(
+ queries={
+ "a": "Trajectory good\nScore: 5",
+ "b": "Trajectory not good\nScore: 1",
+ },
+ sequential_responses=True,
+ )
+ chain = TrajectoryEvalChain.from_llm(llm=llm) # type: ignore
+ res = chain(
+ {
+ "question": "What is your favorite food?",
+ "agent_trajectory": intermediate_steps,
+ "answer": "I like pie.",
+ }
+ )
+ assert res["score"] == 5
+
+ res = chain(
+ {
+ "question": "What is your favorite food?",
+ "agent_trajectory": intermediate_steps,
+ "answer": "I like pie.",
+ "reference": "Paris",
+ }
+ )
+ assert res["score"] == 1
diff --git a/tests/unit_tests/evaluation/comparison/__init__.py b/tests/unit_tests/evaluation/comparison/__init__.py
new file mode 100644
index 0000000000000..e69de29bb2d1d
diff --git a/tests/unit_tests/evaluation/comparison/test_eval_chain.py b/tests/unit_tests/evaluation/comparison/test_eval_chain.py
new file mode 100644
index 0000000000000..9cf4ca8c670dd
--- /dev/null
+++ b/tests/unit_tests/evaluation/comparison/test_eval_chain.py
@@ -0,0 +1,39 @@
+"""Test the comparison chains."""
+
+
+from langchain.evaluation.comparison.eval_chain import PairwiseStringEvalChain
+from tests.unit_tests.llms.fake_llm import FakeLLM
+
+
+def test_pairwise_string_comparison_chain() -> None:
+ llm = FakeLLM(
+ queries={
+ "a": "The values are the same.\n[[C]]",
+ "b": "A is clearly better than b.\n[[A]]",
+ "c": "B is clearly better than a.\n[[B]]",
+ },
+ sequential_responses=True,
+ )
+ chain = PairwiseStringEvalChain.from_llm(llm=llm)
+ res = chain.evaluate_string_pairs(
+ output_a="I like pie.",
+ output_b="I love pie.",
+ input="What is your favorite food?",
+ )
+ assert res["value"] is None
+ assert res["score"] == 0.5
+ assert res["reasoning"] == "The values are the same."
+ res = chain.evaluate_string_pairs(
+ output_a="I like pie.",
+ output_b="I like pie.",
+ input="What is your favorite food?",
+ )
+ assert res["value"] == "A"
+ assert res["score"] == 1
+ res = chain.evaluate_string_pairs(
+ output_a="I like pie.",
+ output_b="I hate pie.",
+ input="What is your favorite food?",
+ )
+ assert res["value"] == "B"
+ assert res["score"] == 0
diff --git a/tests/unit_tests/evaluation/criteria/__init__.py b/tests/unit_tests/evaluation/criteria/__init__.py
new file mode 100644
index 0000000000000..e69de29bb2d1d
diff --git a/tests/unit_tests/evaluation/criteria/test_eval_chain.py b/tests/unit_tests/evaluation/criteria/test_eval_chain.py
new file mode 100644
index 0000000000000..bbe977274ab73
--- /dev/null
+++ b/tests/unit_tests/evaluation/criteria/test_eval_chain.py
@@ -0,0 +1,31 @@
+"""Test the criteria eval chain."""
+
+
+from langchain.evaluation.criteria.eval_chain import (
+ HELPFULNESS_CRITERION,
+ CriteriaEvalChain,
+)
+from langchain.evaluation.schema import StringEvaluator
+from tests.unit_tests.llms.fake_llm import FakeLLM
+
+
+def test_resolve_criteria() -> None:
+ assert CriteriaEvalChain.resolve_criteria("helpfulness") == HELPFULNESS_CRITERION
+ assert CriteriaEvalChain.resolve_criteria(["helpfulness"]) == HELPFULNESS_CRITERION
+
+
+def test_criteria_eval_chain() -> None:
+ chain = CriteriaEvalChain.from_llm(
+ llm=FakeLLM(
+ queries={"text": "The meaning of life\nY"}, sequential_responses=True
+ ),
+ criteria={"my criterion": "my criterion description"},
+ )
+ result = chain.evaluate_strings(
+ prediction="my prediction", reference="my reference", input="my input"
+ )
+ assert result["reasoning"] == "The meaning of life"
+
+
+def test_implements_string_protocol() -> None:
+ assert isinstance(CriteriaEvalChain, StringEvaluator)
diff --git a/tests/unit_tests/evaluation/qa/test_eval_chain.py b/tests/unit_tests/evaluation/qa/test_eval_chain.py
index ac77a97f898cb..514fd28757cce 100644
--- a/tests/unit_tests/evaluation/qa/test_eval_chain.py
+++ b/tests/unit_tests/evaluation/qa/test_eval_chain.py
@@ -4,11 +4,13 @@
import pytest
+from langchain.chains.llm import LLMChain
from langchain.evaluation.qa.eval_chain import (
ContextQAEvalChain,
CotQAEvalChain,
QAEvalChain,
)
+from langchain.evaluation.schema import StringEvaluator
from tests.unit_tests.llms.fake_llm import FakeLLM
@@ -44,3 +46,24 @@ def test_context_eval_chain(chain_cls: Type[ContextQAEvalChain]) -> None:
assert outputs[0] == outputs[1]
assert "text" in outputs[0]
assert outputs[0]["text"] == "foo"
+
+
+@pytest.mark.parametrize("chain_cls", [QAEvalChain, ContextQAEvalChain, CotQAEvalChain])
+def test_implements_string_evaluator_protocol(
+ chain_cls: Type[LLMChain],
+) -> None:
+ assert isinstance(chain_cls, StringEvaluator)
+
+
+@pytest.mark.parametrize("chain_cls", [QAEvalChain, ContextQAEvalChain, CotQAEvalChain])
+def test_returns_expected_results(
+ chain_cls: Type[LLMChain],
+) -> None:
+ fake_llm = FakeLLM(
+ queries={"text": "The meaning of life\nCORRECT"}, sequential_responses=True
+ )
+ chain = chain_cls.from_llm(fake_llm) # type: ignore
+ results = chain.evaluate_strings(
+ prediction="my prediction", reference="my reference", input="my input"
+ )
+ assert results["score"] == 1
diff --git a/tests/unit_tests/evaluation/run_evaluators/__init__.py b/tests/unit_tests/evaluation/run_evaluators/__init__.py
new file mode 100644
index 0000000000000..e69de29bb2d1d
diff --git a/tests/unit_tests/evaluation/run_evaluators/test_implementations.py b/tests/unit_tests/evaluation/run_evaluators/test_implementations.py
new file mode 100644
index 0000000000000..1f5f5a39fd2f2
--- /dev/null
+++ b/tests/unit_tests/evaluation/run_evaluators/test_implementations.py
@@ -0,0 +1,54 @@
+"""Test run evaluator implementations basic functionality."""
+
+from uuid import UUID
+
+import pytest
+from langchainplus_sdk.schemas import Example, Run
+
+from langchain.evaluation.run_evaluators import get_criteria_evaluator, get_qa_evaluator
+from tests.unit_tests.llms.fake_llm import FakeLLM
+
+
+@pytest.fixture
+def run() -> Run:
+ return Run(
+ id=UUID("f77cd087-48f7-4c62-9e0e-297842202107"),
+ name="My Run",
+ inputs={"input": "What is the answer to life, the universe, and everything?"},
+ outputs={"output": "The answer is 42."},
+ start_time="2021-07-20T15:00:00.000000+00:00",
+ end_time="2021-07-20T15:00:00.000000+00:00",
+ run_type="chain",
+ execution_order=1,
+ )
+
+
+@pytest.fixture
+def example() -> Example:
+ return Example(
+ id=UUID("f77cd087-48f7-4c62-9e0e-297842202106"),
+ dataset_id=UUID("f77cd087-48f7-4c62-9e0e-297842202105"),
+ inputs={"input": "What is the answer to life, the universe, and everything?"},
+ outputs={"output": "The answer is 42."},
+ created_at="2021-07-20T15:00:00.000000+00:00",
+ )
+
+
+def test_get_qa_evaluator(run: Run, example: Example) -> None:
+ """Test get_qa_evaluator."""
+ eval_llm = FakeLLM(
+ queries={"a": "This checks out.\nCORRECT"}, sequential_responses=True
+ )
+ qa_evaluator = get_qa_evaluator(eval_llm)
+ res = qa_evaluator.evaluate_run(run, example)
+ assert res.value == "CORRECT"
+ assert res.score == 1
+
+
+def test_get_criteria_evaluator(run: Run, example: Example) -> None:
+ """Get a criteria evaluator."""
+ eval_llm = FakeLLM(queries={"a": "This checks out.\nY"}, sequential_responses=True)
+ criteria_evaluator = get_criteria_evaluator(eval_llm, criteria="conciseness")
+ res = criteria_evaluator.evaluate_run(run, example)
+ assert res.value == "Y"
+ assert res.score == 1
diff --git a/tests/unit_tests/output_parsers/test_base_output_parser.py b/tests/unit_tests/output_parsers/test_base_output_parser.py
index 30166072ca29a..cb44633c67893 100644
--- a/tests/unit_tests/output_parsers/test_base_output_parser.py
+++ b/tests/unit_tests/output_parsers/test_base_output_parser.py
@@ -1,5 +1,6 @@
"""Test the BaseOutputParser class and its sub-classes."""
from abc import ABC
+from collections import defaultdict
from typing import List, Optional, Set, Type
import pytest
@@ -42,12 +43,12 @@ def test_subclass_implements_type(cls: Type[BaseOutputParser]) -> None:
def test_all_subclasses_implement_unique_type() -> None:
- types = []
+ types = defaultdict(list)
for cls in _NON_ABSTRACT_PARSERS:
try:
- types.append(cls._type)
+ types[cls._type].append(cls.__name__)
except NotImplementedError:
# This is handled in the previous test
pass
- dups = set([t for t in types if types.count(t) > 1])
+ dups = {t: names for t, names in types.items() if len(names) > 1}
assert not dups, f"Duplicate types: {dups}"