You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
docs: update docs to reflect the the addition of Libraries and the Hybrid Knowledge Base type (#1853)
* update docs to reflect the the addtion of Libraries and the Hybrid
Knowledge Base type
* fix typo
* update docs to reflect the the addtion of Libraries and the Hybrid
Knowledge Base type
* fix typo
* Incorporate @emjay07 feedback.
* make format
Copy file name to clipboardExpand all lines: docs/griptape-cloud/data-sources/create-data-source.md
+2
Original file line number
Diff line number
Diff line change
@@ -11,6 +11,8 @@ Follow these steps to create a data source. For this example, we will create a d
11
11
1. Enter the URL of a web page that you want to use as a data source, for example https://www.griptape.ai.
12
12
1. Click Create to submit the form and create your data source.
13
13
14
+
## Data Source Types
15
+
14
16
### Web Page
15
17
16
18
You can scrape and ingest a single, public web page by providing a URL. If you wish to scrape multiple pages, you must create multiple Data Sources. However, you can then add all of the pages to the same Knowledge Base if you wish to access all content from the pages together.
Copy file name to clipboardExpand all lines: docs/griptape-cloud/data-sources/getting-started-with-data-sources.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -1,6 +1,6 @@
1
1
# Getting Started with Data Sources
2
2
3
-
In many cases, you need an application to use highly specific and detailed data, data that is very recent or constantly updating, or sensitive data that simply isn't available to LLMs.
3
+
In many cases, you need an application to use highly specific and detailed data, data that is very recent or constantly updating, or sensitive data that simply isn't available to LLMs during training.
All AI-powered software applications require data to perform their jobs. Large language models (LLMs) are already infused with a considerable volume of publicly accessible information, and so a basic chatbot application may not need anything other than a user's input 'prompt' to generate a useful output.
3
+
All AI-powered software applications require data to perform their jobs. A significant amount of information is encoded into Large language models (LLMs) during training, so basic applications may not need anything other than a user's input 'prompt' to generate useful output.
4
4
5
-
Data Sources allow you to bring your own data to Griptape Cloud. By pointing Griptape Cloud at your data, you can make it accessible to your LLM-powered applications.
5
+
Creating Data Sources in Griptape Cloud allow you to bring your own data to use in your AI applications running on Griptape Cloud.
6
6
7
-
Data Sources are the first step to Griptape's retrieval-augemented generation (RAG) pipeline. They allow you to bring your own data to ingest and transform. You can then make one or more Data Source available to your AI applications via [Knowledge Bases](../knowledge-bases/create-knowledge-base.md)
7
+
Data Sources are the first step in creating a Griptape Cloud retrieval-augemented generation (RAG) pipeline. Data Sources allow you to ingest your own data, and optionally to transform it. You can then make one or more Data Sources available to your AI applications either via a [Knowledge Base](../knowledge-bases/create-knowledge-base.md) or a [Retriever](../retrievers/what-are-retrievers.md).
8
8
9
-
Connecting to external data -- by creating a Data Source -- is the first step of building a retrieval-augmented AI application. Once you create a Data Source, you can make it available to your application by adding it to a knowledge base.
10
-
11
-
Griptape Data Sources extract, ingest, and prepare your data so that it can be retrieved and used by LLMs. This is an important step because LLMs work best with data when it is represented in a particular format. These formats often differ from how the information might best be presented to human users or even other software applications. For example, the text of a web page must be cleaned to remove extraneous information, annotated with metadata, segmented into chunks, and converted into vector embeddings before it can be stored in a suitable database.
12
-
13
-
Typically, developers must deploy and operate this process themselves. It can be time consuming, error-prone, and costly. In Griptape Cloud, this process is automated for you.
9
+
Griptape Cloud Data Sources ingest your data and store it in your Data Lake in Griptape Cloud.
Copy file name to clipboardExpand all lines: docs/griptape-cloud/knowledge-bases/accessing-data.md
+1-1
Original file line number
Diff line number
Diff line change
@@ -4,7 +4,7 @@ You can `Search` or `Query` the Knowledge Base for information contained in your
4
4
5
5
## From the Cloud Console
6
6
7
-
You can explore your data with a natural language question on the `Test` tab of your Knowledge Base. Compare and contrast the results of `Search` vs. `Query`to understand which is correct for your application.
7
+
You can explore your data with a natural language question on the `Query` tab of your Knowledge Base. You can use *+ Add Schema Argument* to add schema arguments to query structured data in a Hybrid Knowledge Base. Selecting a *count* schema argument allows you to specify the number of results that you wish to return in the response to your query.
Knowledge Bases are the way to organize and access your data ingested from [Data Sources](../data-sources/create-data-source.md). You can specify multiple Data Sources per Knowledge Base in order to access data ingested from different sources all in one place.
3
+
Knowledge Bases are way to organize and access your data ingested from [Data Sources](../data-sources/create-data-source.md). You can specify multiple Data Sources per Knowledge Base in order to access data ingested from different sources all in one place.
4
4
5
-
## Create a Knowledge Base
5
+
## Creating a Vector Knowledge Base
6
6
7
-
You can [create a Knowledge Base in the Griptape Cloud console](https://cloud.griptape.ai/knowledge-bases/create) by specifying which Data Sources you wish to include. Once created, you can [access your data](accessing-data.md).
7
+
You can create a Griptape Cloud Vector Knowledge Base on the [Create Knowledge Base](https://cloud.griptape.ai/knowledge-bases/create) page in Griptape Cloud console. When doing so, you will asked provide a name and to select the Data Sources that you wish to include in your new Knowledge Base. Once created, you can [access your data](accessing-data.md).
8
+
9
+
If you wish to provide your own Postgres & pgvector endpoint, you will be prompted to provide your database connection string and password, as well as to select the Data Sources that you wish to use to populate this Knowledge Base.
10
+
11
+
## Creating a Hybrid Knowledge Base
12
+
13
+
To create a Griptape Cloud Hybrid Knowledge Base, select the Griptape Cloud card in the Hybrid Knowledge Base section on the [Create Knowledge Base](https://cloud.griptape.ai/knowledge-bases/create) page in Griptape Cloud console. You will be prompted to provide a name and to select the Data Sources that you wish to include in your Knowledge Base. The final step is the verify the field mappings and confirm which columns are structured columns and which columns contain unstructued data. We attempt to determine which columns are of which type, but you can modify the mappings if you see any columns that have been incorrectly mapped.
Follow these steps to create a Knowledge Base. For this example, we will use the fully-managed Griptape Cloud database. EDB customers can select the pgvector option.
5
+
Follow these steps to create a Vector Knowledge Base. For this example, we will use the fully-managed Griptape Cloud vector store. If you are using EDB Postgres or another type of Postgres instance as your vector store, select the pgvector option.
6
6
7
7
1. Navigate to the [Knowledge Bases](https://cloud.griptape.ai/knowledge-bases) page in the Griptape Cloud Console.
8
8
1. Click *Create Knowledge Base*.
9
-
1. Select the Griptape Cloud Knowledge Base type.
9
+
1. Select the Griptape Cloud Vector Knowledge Base type.
10
10
1. Give your Knowledge Base a name and a description (optional).
11
11
1. Select the Data Source(s) you want to include in the Knowledge Base.
12
12
1. Click *Create* to submit the form.
13
13
14
14
> ### Pro Tip:
15
15
>
16
-
> You can add a Data Source to as many Knowledge Bases (KBs) as you want. For example, you can include a Frequently Asked Questions Data Source in both a New Employee Onboarding KB and a Customer Support Playbook KB.
16
+
> You can add a Data Source to as many Vector Knowledge Bases (KBs) as you want. For example, you can include a Frequently Asked Questions Data Source in both a New Employee Onboarding KB and a Customer Support Playbook KB.
17
17
18
18
## What's happening?
19
19
20
-
Once you have created the Knowledge Base, we will automatically begin the process of upserting your data to a database. This process is known as a Knowledge Base job. It typically takes just a few moments.
20
+
Once you have created the Vector Knowledge Base, we will automatically begin the process of upserting your data to a database. This process is known as a Knowledge Base job. It typically takes just a few moments.
21
21
22
22
While the job is in progress, you will be directed to the Knowledge Base detail page where you can observe the job status as well as view and edit Knowledge Base details such as the name, description, and Data Sources to be included.
23
23
24
-
## How to use a Knowledge Base
24
+
## How to use a Vector Knowledge Base
25
25
26
-
When your Knowledge Base is ready, the data it contains becomes available for applications to retrieve via Griptape Assistants, or Structures such as Agents.
26
+
When your Vector Knowledge Base is ready, the data it contains becomes available for applications to retrieve via Griptape Assistants, or Structures such as Agents.
27
27
28
28
> ### Pro Tip:
29
29
>
30
-
> You can perform a test query by selecting the Query tab and entering some information that you know is in your data. The result will be a 'raw' response that contains the embedded text and other query parameters. This feature is useful for quick testing and debugging.
30
+
> You can perform a test query by selecting the Query tab and entering some information that you know is in your data. The result will be a 'raw' response that contains the embedded text and other query parameters. This feature is useful for testing and debugging.
31
31
32
-
The next step of using your Knowledge Base is connecting it to an application, such as a chat Assistant, that can retrieve your data and use it to generate useful responses.
32
+
The next step of using your Vector Knowledge Base is connecting it to an application, such as a chat Assistant, that can retrieve your data and use it to generate useful responses.
33
33
34
34
Follow these steps to create and use a simple chat Assistant.
35
35
36
36
1. Navigate to the [Assistants](https://cloud.griptape.ai/assistants) page in the Griptape Cloud Console.
37
37
1. Click *Create Assistant*.
38
38
1. Give your Assistant a name and description (optional).
39
-
1. Select your Knowledge Base from the *Knowledge Bases* dropdown menu.
39
+
1. Select your Vector Knowledge Base from the *Knowledge Bases* dropdown menu.
40
40
1. Click *Create* to submit the form.
41
41
42
42
You will be directed to the Assistant chat screen. Type a message to start a conversation thread with your new Assistant. Try asking it something about your data!
@@ -45,12 +45,4 @@ You will be directed to the Assistant chat screen. Type a message to start a con
45
45
>
46
46
> Add a Ruleset to provide your Assistant with instructions and guidance for how it should behave.
47
47
48
-
## Types of Knowledge Base engine
49
-
50
-
Knowledge Bases are stored and accessed in various ways, according to the data types they contain and where the Data Sources are located. For example, unstructured text data is typically stored in a vector database. Griptape Cloud provides a fully-managed vector database, so that you don't need to deploy and operate it yourself. However, some customers have existing database installations where they would like their Knowledge Bases to reside.
51
-
52
-
###Griptape Cloud
53
-
Upsert your data to a fully-managed Griptape Cloud database.
54
-
55
-
###PostgreSQL with pgvector
56
-
Upsert your data to your own PostgreSQL pgvector database by providing the connection string and password for authorization.
48
+
For more information about different Knowledge Base types, see [Knowledge Base Types](./knowledge-base-types.md)
Griptape Cloud supports two different types of Knowledge Base:
4
+
5
+
## Vector Knowledge Bases
6
+
7
+
Vector Knowledge Bases generate embeddings from the data in the specified data source or data sources and upsert this, together with the original data, into a database that acts as a vector store. Griptape Cloud has a built-in fully-managed vector store. You can also provide your own Postgres & pgvector endpoint and configure Griptape Cloud to use this as the vector store for your Knowledge Base.
8
+
9
+
Queries made against Vector Knowledge Bases are vector search queries.
10
+
11
+
## Hybrid Knowledge Bases
12
+
13
+
Hybrid Knowledge Bases also generate embeddings for unstructured data. They differ from Vector Knowledge Bases in that they are also able to store structured data alongside vectors and unstructured data.
14
+
15
+
Griptape Cloud currently supports the creation of Hybrid Knowledge Bases from CSV data (via the Amazon S3, Google Drive & Griptape Cloud Data Lake Data Source types), and from Google Sheets through the Google Drive Data Source type.
16
+
17
+
Queries made against Hybrid Knowledge Bases can be SQL queries made against structured fields, vector search queries made against unstructured fields, or a combination of the two query types.
Griptape Knowledge Bases are collections of Data Sources that applications can query to retrieve information. They give you control over which Data Sources can be accessed, as well as how the data can be retrieved. Once you have created the Data Source(s) you need, simply add them to a Knowledge Base so that your application, such as a Griptape Assistant or Agent, can access it.
3
+
Knowledge Bases are collections of Data Sources that applications can query to retrieve information. Knowledge Bases enable you control which Data Sources can be accessed by applications. Once you have created the Data Source(s) you need, add them to a Knowledge Base so that your application, such as a Griptape Cloud Assistant or an Agent in a Structure, can access them.
4
4
5
-
When you add Data Sources to a Knowledge Base, your data is upserted to a database that is optimized for LLMs to retrieve quickly and process efficiently. Typically, this requires maintaining a database to store your data, operating a data ingestion (ETL) pipeline to collect it, and vending a query endpoint to make it available. Griptape Cloud automates this process for you.
5
+
When you add Data Sources to a Knowledge Base, your data is automatically upserted to a database that is optimized for use with LLMs so that the data can be retrieved quickly and processed efficiently. Achieving this outside Griptape Cloud requires you to maintain a database to store your data, operating a data ingestion pipeline to collect data, and provide and operate a query endpoint to make it available. Griptape Cloud Knowledge Bases automate this process for you.
Retrievers provide retrieval augmented generation capabilities within Griptape Cloud. They are a fully-managed implementation of the [RAG Engine](../../griptape-framework/engines/rag-engines.md) within the [Griptape Framework](../../griptape-framework/index.md).
4
+
5
+
Using a Retriever will give more accurate and effective results than using a Knowledge Base because Retrievers provide reranking capabilities. They also support queries across multiple Knowledge Bases and apply reranking across results results returned from multiple Knowledge Bases, which is a powerful capability that is required for many RAG use-cases.
6
+
7
+
Retrievers include a standard set of modules known as Retriever Components (to avoid confusion with the RAG modules in Griptape Framework). Retreiver Components are used to implement the different stages of the RAG pipeline.
8
+
9
+
## Response Component Configuration
10
+
11
+
If you are using your Retriever with a Griptape Cloud Assistant, set the *Response**Type* to *Text Chunks*. If your use-case requires a natural language response, set the *Response**Type* to *Prompt with Rulesets* and specify which *Rulesets* you wish to apply.
0 commit comments