forked from fkzhao/ragflow
-
Notifications
You must be signed in to change notification settings - Fork 0
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
Initial draft of configure knowledge base (infiniflow#794)
### What problem does this PR solve? _Briefly describe what this PR aims to solve. Include background context that will help reviewers understand the purpose of the PR._ ### Type of change - [x] Documentation Update
- Loading branch information
1 parent
c6c9dbd
commit d54d137
Showing
2 changed files
with
133 additions
and
3 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,132 @@ | ||
| # Configure a knowledge base | ||
|
|
||
| Knowledge base, hallucination-free chat, and file management are three pillars of RAGFlow. RAGFlow's AI chats are based on knowledge bases. Each of RAGFlow's knowledge bases serves as a knowledge source, *parsing* files uploaded from your local machine and file references generated in **File Management** into the real 'knowledge' for future AI chats. This guide demonstrates some basic usages of the knowledge base feature, covering the following topics: | ||
|
|
||
| - Create a knowledge base | ||
| - Configure a knowledge base | ||
| - Search for a knowledge base | ||
| - Delete a knowledge base | ||
|
|
||
| ## Create knowledge base | ||
|
|
||
| With multiple knowledge bases, you can build more flexible, diversified question answering. To create your first knowledge base: | ||
|
|
||
|  | ||
|
|
||
| _Each time a knowledge base is created, a folder with the same name is generated in the **root/.knowledgebase** directory._ | ||
|
|
||
| ## Configure knowledge base | ||
|
|
||
| The following screen shot shows the configuration page of a knowledge base. A proper configuration of your knowledge base is crucial for future AI chats. For example, choosing the wrong embedding model or chunk method would cause unexpected semantic loss or mismatched answers in chats. | ||
|
|
||
|  | ||
|
|
||
| This section covers the following topics: | ||
|
|
||
| - Select chunk method | ||
| - Select embedding model | ||
| - Upload file | ||
| - Parse file | ||
| - Intervene with file parsing results | ||
| - Run retrieval testing | ||
|
|
||
| ### Select chunk method | ||
|
|
||
| RAGFlow offers multiple chunking template to facilitate chunking files of different layouts and ensure semantic integrity. In **Chunk method**, you can choose the default template that suits the layouts and formats of your files. The following table shows the descriptions and the compatible file formats of each supported chunk template: | ||
|
|
||
| | **Template** | Description | File format | | ||
| | ------------ | ------------------------------------------------------------ | ---------------------------------------------------- | | ||
| | General | Files are consecutively chunked based on a preset chunk token number. | DOCX, EXCEL, PPT, PDF, TXT, JPEG, JPG, PNG, TIF, GIF | | ||
| | Q&A | | EXCEL, CSV/TXT | | ||
| | Manual | | PDF | | ||
| | Table | | EXCEL, CSV/TXT | | ||
| | Paper | | PDF | | ||
| | Book | | DOCX, PDF, TXT | | ||
| | Laws | | DOCX, PDF, TXT | | ||
| | Presentation | | PDF, PPTX | | ||
| | Picture | | JPEG, JPG, PNG, TIF, GIF | | ||
| | One | The entire document is chunked as one. | DOCX, EXCEL, PDF, TXT | | ||
|
|
||
| You can also change the chunk template for a particular file on the **Datasets** page. | ||
|
|
||
|  | ||
|
|
||
| ### Select embedding model | ||
|
|
||
| An embedding model builds vector index on file chunks. Once you have chosen an embedding model and used it to parse a file, you are no longer allowed to change it. To switch to a different embedding model, you *must* deletes all completed file chunks in the knowledge base. The obvious reason is that we must *ensure* that all files in a specific knowledge base are parsed using the *same* embedding model (ensure that they are compared in the same embedding space). | ||
|
|
||
| The following embedding models can be deployed locally: | ||
|
|
||
| - BAAI/bge-base-en-v1.5 | ||
| - BAAI/bge-large-en-v1.5 | ||
| - BAAI/bge-small-en-v1.5 | ||
| - BAAI/bge-small-zh-v1.5 | ||
| - jinaai/jina-embeddings-v2-base-en | ||
| - jinaai/jina-embeddings-v2-small-en | ||
| - nomic-ai/nomic-embed-text-v1.5 | ||
| - sentence-transformers/all-MiniLM-L6-v2 | ||
| - maidalun1020/bce-embedding-base_v1 | ||
|
|
||
| ### Upload file | ||
|
|
||
| - RAGFlow's **File Management** allows you to link a file to multiple knowledge bases, in which case each target knowledge base holds a reference to the file. | ||
| - In **Knowledge Base**, you are also given the option of uploading a single file or a folder of files (bulk upload) from your local machine to a knowledge base, in which case the knowledge base holds file copies. | ||
|
|
||
| While uploading files directly to a knowledge base seems more convenient, we *highly* recommend uploading files to **File Management** and then linking them to the target knowledge bases. This way, you can avoid permanently deleting files uploaded to the knowledge base. | ||
|
|
||
| ### Parse file | ||
|
|
||
| File parsing is a crucial topic in knowledge base configuration. The meaning of file parsing in RAGFlow is twofold: chunking files based on file layout and building embedding and full-text (keyword) indexes on these chunks. After having selected the chunk method and embedding model, you can start parsing an file: | ||
|
|
||
|  | ||
|
|
||
| - Click the play button next to **UNSTART** to start file parsing. | ||
| - Click the red-cross icon and then refresh, if your file parsing stalls for a long time. | ||
| - As shown above, RAGFlow allows you to use a different chunk method for a particular file, offering flexibility beyond the default method. | ||
| - As shown above, RAGFlow allows you to enable or disable individual files, offering finer control over knowledge base-based AI chats. | ||
|
|
||
| ### Intervene with file parsing results | ||
|
|
||
| RAGFlow features visibility and explainability, allowing you to view the chunking results and intervene where necessary. To do so: | ||
|
|
||
| 1. Click on the file that completes file parsing to view the chunking results: | ||
|
|
||
| _You are taken to the **Chunk** page:_ | ||
|
|
||
|  | ||
|
|
||
| 2. Hover over each snapshot for a quick view of each chunk. | ||
|
|
||
| 3. Double click the chunked texts to add keywords or make *manual* changes where necessary: | ||
|
|
||
|  | ||
|
|
||
| 4. In Retrieval testing, ask a quick question in **Test text** to double check if your configurations work: | ||
|
|
||
| _As you can tell from the following, RAGFlow responds with truthful citations._ | ||
|
|
||
|  | ||
|
|
||
| ### Run retrieval testing | ||
|
|
||
| RAGFlow uses multiple recall of both full-text search and vector search in its chats. Prior to setting up an AI chat, consider adjusting the following parameters to ensure that the intended information always turns up in answers: | ||
|
|
||
| - Similarity threshold: Chunks with similarities below the threshold will be filtered. Defaultly set to 0.2. | ||
| - Vector similarity weight: The percentage by which vector similarity contributes to the overall score. Defaultly set to 0.3. | ||
|
|
||
|  | ||
|
|
||
| ## Search for knowledge base | ||
|
|
||
| As of RAGFlow v0.5.0, the search feature is still in a rudimentary form, supporting only knowledge base search by name. | ||
|
|
||
|  | ||
|
|
||
| ## Delete knowledge base | ||
|
|
||
| You are allowed to delete a knowledge base. Hover your mouse over the three dot of the intended knowledge base card and the **Delete** option appears. Once you delete a knowledge base, the associated folder under **root/.knowledge** directory is AUTOMATICALLY REMOVED. The consequence is: | ||
|
|
||
| - The files uploaded directly to the knowledge base are gone; | ||
| - The file references, which you created from within **File Management**, are gone, but the associated files still exist in **File Management**. | ||
|
|
||
|  |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters