Skip to content

Commit

Permalink
Squashed commit of the following:
Browse files Browse the repository at this point in the history
commit 673a28e
Author: KevinHuSh <[email protected]>
Date:   Fri May 17 20:03:00 2024 +0800

    fix bug of chat without stream (infiniflow#830)

    ### What problem does this PR solve?

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 2bfacd0
Author: KevinHuSh <[email protected]>
Date:   Fri May 17 18:06:20 2024 +0800

    refine doc about API: completion (infiniflow#829)

    ### What problem does this PR solve?
    infiniflow#808

    ### Type of change

    - [x] Documentation Update

commit b3c923d
Author: KevinHuSh <[email protected]>
Date:   Fri May 17 17:51:54 2024 +0800

    add doc ids in API: completion (infiniflow#827)

    ### What problem does this PR solve?
    infiniflow#808

    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit a1586e0
Author: KevinHuSh <[email protected]>
Date:   Fri May 17 17:27:39 2024 +0800

    correct mismatched kb doc number (infiniflow#826)

    ### What problem does this PR solve?

    infiniflow#620

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit f6a5994
Author: KevinHuSh <[email protected]>
Date:   Fri May 17 17:07:33 2024 +0800

    fix zhipuAI stream issue (infiniflow#825)

    ### What problem does this PR solve?

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 081f922
Author: GYH <[email protected]>
Date:   Fri May 17 15:58:05 2024 +0800

    0517 list chunks (infiniflow#821)

    ### What problem does this PR solve?

    infiniflow#717

    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit 9f0f5b4
Author: balibabu <[email protected]>
Date:   Fri May 17 15:38:28 2024 +0800

    Default language will be given according to the browse setting and also can be configured infiniflow#801 (infiniflow#823)

    ### What problem does this PR solve?

    Default language will be given according to the browse setting and also
    can be configured infiniflow#801
    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit a2a6a35
Author: KevinHuSh <[email protected]>
Date:   Fri May 17 15:35:09 2024 +0800

    fix doc number miss-match issue (infiniflow#822)

    ### What problem does this PR solve?

    infiniflow#620

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 9e5d501
Author: KevinHuSh <[email protected]>
Date:   Fri May 17 14:33:19 2024 +0800

    fix data init error (infiniflow#820)

    ### What problem does this PR solve?

    infiniflow#810

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 4ca176b
Author: balibabu <[email protected]>
Date:   Fri May 17 14:16:55 2024 +0800

    fix: thumbnails are too large in the chat box infiniflow#818 (infiniflow#819)

    ### What problem does this PR solve?

    fix: thumbnails are too large in the chat box infiniflow#818

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit c3bc72d
Author: KevinHuSh <[email protected]>
Date:   Fri May 17 14:04:21 2024 +0800

    fix too large thumbnail issue (infiniflow#817)

    ### What problem does this PR solve?

    infiniflow#709

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 2dd705f
Author: LiYongHui <[email protected]>
Date:   Fri May 17 13:47:05 2024 +0800

    feat: add feishu oauth (infiniflow#815)

    ### What problem does this PR solve?

    The back-end code adds Feishu oauth

    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

    Co-authored-by: yonghui li <[email protected]>

commit d161410
Author: KevinHuSh <[email protected]>
Date:   Fri May 17 12:07:00 2024 +0800

    fix stream chat for ollama (infiniflow#816)

    ### What problem does this PR solve?

    infiniflow#709

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 05fa3ae
Author: KevinHuSh <[email protected]>
Date:   Fri May 17 09:00:24 2024 +0800

    use smaller docker images (infiniflow#813)

    ### What problem does this PR solve?

    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit e73ce39
Author: KevinHuSh <[email protected]>
Date:   Fri May 17 08:51:29 2024 +0800

    Add 2 embeding models from OpenAI (infiniflow#812)

    ### What problem does this PR solve?

    infiniflow#810

    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit d54d137
Author: writinwaters <[email protected]>
Date:   Thu May 16 21:27:09 2024 +0800

    Initial draft of configure knowledge base (infiniflow#794)

    ### What problem does this PR solve?

    _Briefly describe what this PR aims to solve. Include background context
    that will help reviewers understand the purpose of the PR._

    ### Type of change

    - [x] Documentation Update

commit c6c9dbd
Author: balibabu <[email protected]>
Date:   Thu May 16 20:15:02 2024 +0800

    feat: Support for conversational streaming (infiniflow#809)

    ### What problem does this PR solve?

    feat: Support for conversational streaming
    infiniflow#709

    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit 95f8091
Author: KevinHuSh <[email protected]>
Date:   Thu May 16 20:14:53 2024 +0800

    add stream chat (infiniflow#811)

    ### What problem does this PR solve?

    infiniflow#709
    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit d6772f5
Author: KevinHuSh <[email protected]>
Date:   Thu May 16 16:17:48 2024 +0800

    add version (infiniflow#807)

    ### What problem does this PR solve?
    infiniflow#709
    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit 63ca15c
Author: lcjia_you <[email protected]>
Date:   Thu May 16 10:49:41 2024 +0800

    Fix a bug in 'assistant-setting.tsx' that causes the upload button to… (infiniflow#796)

    … incorrectly appear on the model settings page.

    ### What problem does this PR solve?

    This is an issue with the Upload component on the assistant-setting
    page. I use the show variable to explicitly control the button component
    within it.

    see:

    ![20240516000417](https://github.com/infiniflow/ragflow/assets/37476944/de88f911-6dbd-412d-a981-86cf60aa2257)

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)
    - [x] Other (please describe): Add the local models that DeepDoc depends
    on to the gitignore file in dev mode.

    Signed-off-by: liuchao <[email protected]>

commit 7b144cc
Author: KevinHuSh <[email protected]>
Date:   Thu May 16 09:10:29 2024 +0800

    fix: can't capitalize file or folder name (infiniflow#798)

    ### What problem does this PR solve?

    infiniflow#792

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 1c4e92e
Author: KevinHuSh <[email protected]>
Date:   Thu May 16 09:00:12 2024 +0800

    Knowledge base search is case sensitive (infiniflow#797)

    ### What problem does this PR solve?
    infiniflow#793
    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 10e83f2
Author: writinwaters <[email protected]>
Date:   Wed May 15 20:02:41 2024 +0800

    Added file management guide (infiniflow#788)

    ### What problem does this PR solve?

    Added guide with instructions on managing files in RAGFlow.

    ### Type of change

    - [x] Documentation Update

commit 6ff63ee
Author: dashi6174 <[email protected]>
Date:   Wed May 15 16:34:28 2024 +0800

    Support for code files parse (infiniflow#789)

    ### What problem does this PR solve?

    _Briefly describe what this PR aims to solve. Include background context
    that will help reviewers understand the purpose of the PR._

    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit 12b4c56
Author: GYH <[email protected]>
Date:   Wed May 15 16:33:28 2024 +0800

    Updated conversation_api.md document/upload (infiniflow#787)

    ### What problem does this PR solve?

    Updated conversation_api.md document/upload parameter description

    ### Type of change

    - [x] Documentation Update

commit baad35d
Author: balibabu <[email protected]>
Date:   Wed May 15 14:53:36 2024 +0800

    fix: .knowledgebase folder can be deleted bug and change "Add file to knowledge base" to "Link file to knowledge base" bug (infiniflow#786)

    ### What problem does this PR solve?
    fix: .knowledgebase folder can be deleted bug
    fix: change "Add file to knowledge base" to "Link file to knowledge
    base" bug
    infiniflow#783 infiniflow#784

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 5effbfa
Author: balibabu <[email protected]>
Date:   Wed May 15 13:58:30 2024 +0800

    fix: remove Top K in retrieval testing  infiniflow#770 and if the document parsing fails, the error message returned by the backend is displayed (infiniflow#782)

    ### What problem does this PR solve?

    fix: remove Top K in retrieval testing  infiniflow#770
    fix: if the document parsing fails, the error message returned by the
    backend is displayed.

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 4d47b2b
Author: KevinHuSh <[email protected]>
Date:   Wed May 15 13:02:31 2024 +0800

    fix a string format error (infiniflow#781)

    ### What problem does this PR solve?

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit d8c080e
Author: KevinHuSh <[email protected]>
Date:   Wed May 15 12:51:57 2024 +0800

    fix bugs in searching file using keywords (infiniflow#780)

    ### What problem does this PR solve?

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 626ace8
Author: GYH <[email protected]>
Date:   Wed May 15 12:22:11 2024 +0800

    Updated document upload method (infiniflow#777)

    ### What problem does this PR solve?

    api_app.py
    /document/upload
    add two non mandatory parameters
    parser_id:
    [naive,qaresume,manual,table,paper,book,laws,presentation,picture,one]
    run: 1

    ### Type of change
    - [x] New Feature (non-breaking change which adds functionality)

commit 1e923f1
Author: Jin Hai <[email protected]>
Date:   Wed May 15 12:08:32 2024 +0800

    Update README (infiniflow#779)

    ### What problem does this PR solve?

    infiniflow#771

    ### Type of change

    - [x] Documentation Update

    ---------

    Signed-off-by: Jin Hai <[email protected]>

commit 234afb2
Author: balibabu <[email protected]>
Date:   Wed May 15 11:34:57 2024 +0800

    feat: support GPT-4o  infiniflow#771 and hide the add button when the folder is a knowledge base (infiniflow#775)

    ### What problem does this PR solve?

    feat: support GPT-4o  infiniflow#771

    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit aa1c915
Author: KevinHuSh <[email protected]>
Date:   Wed May 15 11:16:08 2024 +0800

    support gpt-4o (infiniflow#773)

    ### What problem does this PR solve?
    infiniflow#771

    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit 77b1520
Author: Jin Hai <[email protected]>
Date:   Wed May 15 10:48:42 2024 +0800

    Refactor message output format (infiniflow#772)

    ### What problem does this PR solve?

    _Briefly describe what this PR aims to solve. Include background context
    that will help reviewers understand the purpose of the PR._

    ### Type of change

    - [x] Refactoring

    Signed-off-by: Jin Hai <[email protected]>

commit 6b06cce
Author: writinwaters <[email protected]>
Date:   Tue May 14 18:46:39 2024 +0800

    Miscellaneous updates (infiniflow#769)

    ### What problem does this PR solve?

    _Briefly describe what this PR aims to solve. Include background context
    that will help reviewers understand the purpose of the PR._

    ### Type of change

    - [x] Documentation Update

commit 282f085
Author: balibabu <[email protected]>
Date:   Tue May 14 16:53:32 2024 +0800

    fix: hide the add button when the folder is a knowledge base (infiniflow#765)

    ### What problem does this PR solve?

    infiniflow#764 fix: hide the add button when the folder is a knowledge base

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit d7744f5
Author: Jin Hai <[email protected]>
Date:   Tue May 14 14:48:15 2024 +0800

    Refactor method name (infiniflow#760)

    ### What problem does this PR solve?

    infiniflow#757

    ### Type of change

    - [x] Refactoring

    Signed-off-by: Jin Hai <[email protected]>

commit 9b21b66
Author: writinwaters <[email protected]>
Date:   Tue May 14 12:22:33 2024 +0800

    Create quickstart.md (infiniflow#743)

    ### What problem does this PR solve?

    Draft quickstart.

    ### Type of change

    - [x] Documentation Update

commit aa03dfa
Author: KevinHuSh <[email protected]>
Date:   Mon May 13 14:02:38 2024 +0800

    fix bug of get file (infiniflow#746)

    ### What problem does this PR solve?

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 69b7c61
Author: Sawyer <[email protected]>
Date:   Mon May 13 09:25:45 2024 +0800

    fix: typo in user_app.py (infiniflow#740)

    ### What problem does this PR solve?

    _Briefly describe what this PR aims to solve. Include background context
    that will help reviewers understand the purpose of the PR._

    ### Type of change

    - [x] Bug Fix (non-breaking change
    - [x] Other (please describe): Fix typo

commit 8769619
Author: Jin Hai <[email protected]>
Date:   Sun May 12 13:40:47 2024 +0800

    Update readme (infiniflow#741)

    ### What problem does this PR solve?

    Update readme.

    ### Type of change

    - [x] Documentation Update

    Signed-off-by: Jin Hai <[email protected]>

commit ffe5737
Author: KevinHuSh <[email protected]>
Date:   Sat May 11 19:47:53 2024 +0800

    let index be batchly. (infiniflow#733)

    ### What problem does this PR solve?

    let index be batchly.

    ### Type of change

    - [x] Refactoring

commit 04a9e95
Author: KevinHuSh <[email protected]>
Date:   Sat May 11 16:04:28 2024 +0800

    let file in knowledgebases visible in file manager (infiniflow#714)

    ### What problem does this PR solve?

    Let file in knowledgebases visible in file manager.
    infiniflow#162

    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit 91b4a18
Author: balibabu <[email protected]>
Date:   Sat May 11 16:03:07 2024 +0800

    Make the app name configurable even after the project is built (infiniflow#731)

    ### What problem does this PR solve?

    Make the app name configurable even after the project is built infiniflow#730

    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit 33eaf6f
Author: Ikko Eltociear Ashimine <[email protected]>
Date:   Fri May 10 12:22:40 2024 +0900

    docs: update README_ja.md (infiniflow#707)

    ### What problem does this PR solve?

    _Briefly describe what this PR aims to solve. Include background context
    that will help reviewers understand the purpose of the PR._

    ### Type of change

    - [x] Documentation Update

commit d65ba3e
Author: balibabu <[email protected]>
Date:   Fri May 10 10:38:39 2024 +0800

    feat: delete the added model infiniflow#503 and display an error message when the requested file fails to parse infiniflow#684  (infiniflow#708)

    ### What problem does this PR solve?

    feat: delete the added model infiniflow#503
    feat: display an error message when the requested file fails to parse
    infiniflow#684

    ### Type of change

    - [x] New Feature (non-breaking change which adds functionality)

commit bef1bbd
Author: CKLogic <[email protected]>
Date:   Fri May 10 09:48:50 2024 +0800

    Update README with Detailed WebUI Service Launch Instructions (infiniflow#694)

    ### What problem does this PR solve?

    Improve README by detailing Launch Service from Source section

    This commit enhances the README document by adding comprehensive steps
    for running the WebUI service in the 'Launch Service from Source'
    section. It aims to provide clearer guidance for users attempting to
    start the service from the source code, making the setup process more
    accessible and understandable.

    Key changes include:
    - Detailed instructions for setting up and running the WebUI service.
    - Necessary prerequisites for launching the service from source.

    This update ensures that users have all the information they need to
    successfully launch the service, improving the overall usability of our
    project.

    ### Type of change

    - [x] Documentation Update

commit 6b36f31
Author: writinwaters <[email protected]>
Date:   Fri May 10 09:48:24 2024 +0800

    Minor editorial updates (infiniflow#700)

    ### What problem does this PR solve?

    Editorial updates only.

    ### Type of change

    - [x] Documentation Update

commit 648a2ba
Author: KevinHuSh <[email protected]>
Date:   Thu May 9 15:32:24 2024 +0800

    fix disabled doc is still retreivalable (infiniflow#695)

    ### What problem does this PR solve?

    Fix that disabled doc is still retreivalable

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 9392b8b
Author: writinwaters <[email protected]>
Date:   Thu May 9 12:37:45 2024 +0800

    0509 faq (infiniflow#693)

    ### What problem does this PR solve?

    Editorial updates only.

    ### Type of change

    - [x] Documentation Update

commit 4153a36
Author: KevinHuSh <[email protected]>
Date:   Thu May 9 11:35:08 2024 +0800

    truncate text to fitin embedding model (infiniflow#692)

    ### What problem does this PR solve?

    ### Type of change

    - [x] Refactoring

commit bca63ad
Author: GYH <[email protected]>
Date:   Thu May 9 11:32:36 2024 +0800

    Update faq.md (infiniflow#685)

    ### What problem does this PR solve?

    Updated FAQ: How to upgrade RAGFlow

    ### Type of change

    - [x] Documentation Update

commit 793e29f
Author: balibabu <[email protected]>
Date:   Thu May 9 11:30:15 2024 +0800

    fix: fix uploaded file time error infiniflow#680 (infiniflow#690)

    ### What problem does this PR solve?

    fix: fix uploaded file time error infiniflow#680
    feat: support preview of word and excel infiniflow#684

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 99be226
Author: KevinHuSh <[email protected]>
Date:   Wed May 8 20:00:14 2024 +0800

    fix coordinate error (infiniflow#686)

    ### What problem does this PR solve?

    infiniflow#683

    ### Type of change

    - [x] Bug Fix (non-breaking change which fixes an issue)

commit 7ddb2f1
Author: KevinHuSh <[email protected]>
Date:   Wed May 8 15:20:45 2024 +0800

    make sure to raise exception if redis is not there (infiniflow#674)

    ### What problem does this PR solve?

    ### Type of change

    - [x] Refactoring

commit c28f7b5
Author: KevinHuSh <[email protected]>
Date:   Wed May 8 13:58:41 2024 +0800

    make sure  the error will be recorded. (infiniflow#672)

    ### What problem does this PR solve?

    ### Type of change

    - [x] Refactoring
  • Loading branch information
fkzhao committed May 20, 2024
1 parent 48cee1f commit a0818bf
Show file tree
Hide file tree
Showing 81 changed files with 5,723 additions and 2,798 deletions.
3 changes: 2 additions & 1 deletion Dockerfile
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM swr.cn-north-4.myhuaweicloud.com/infiniflow/ragflow-base:v1.0
FROM infiniflow/ragflow-base:v2.0
USER root

WORKDIR /ragflow
Expand All @@ -15,6 +15,7 @@ ENV PYTHONPATH=/ragflow/
ENV HF_ENDPOINT=https://hf-mirror.com

ADD docker/entrypoint.sh ./entrypoint.sh
ADD docker/.env ./
RUN chmod +x ./entrypoint.sh

ENTRYPOINT ["./entrypoint.sh"]
2 changes: 1 addition & 1 deletion Dockerfile.cuda
Original file line number Diff line number Diff line change
@@ -1,4 +1,4 @@
FROM swr.cn-north-4.myhuaweicloud.com/infiniflow/ragflow-base:v1.0
FROM FROM infiniflow/ragflow-base:v2.0
USER root

WORKDIR /ragflow
Expand Down
15 changes: 14 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,19 @@

## 💡 What is RAGFlow?

[RAGFlow](https://demo.ragflow.io) is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding. It offers a streamlined RAG workflow for businesses of any scale, combining LLM (Large Language Models) to provide truthful question-answering capabilities, backed by well-founded citations from various complex formatted data.
[RAGFlow](https://ragflow.io/) is an open-source RAG (Retrieval-Augmented Generation) engine based on deep document understanding. It offers a streamlined RAG workflow for businesses of any scale, combining LLM (Large Language Models) to provide truthful question-answering capabilities, backed by well-founded citations from various complex formatted data.

## 📌 Latest Updates

- 2024-05-15 Integrates OpenAI GPT-4o.
- 2024-05-08 Integrates LLM DeepSeek-V2.
- 2024-04-26 Adds file management.
- 2024-04-19 Supports conversation API ([detail](./docs/conversation_api.md)).
- 2024-04-16 Integrates an embedding model 'bce-embedding-base_v1' from [BCEmbedding](https://github.com/netease-youdao/BCEmbedding), and [FastEmbed](https://github.com/qdrant/fastembed), which is designed specifically for light and speedy embedding.
- 2024-04-11 Supports [Xinference](./docs/xinference.md) for local LLM deployment.
- 2024-04-10 Adds a new layout recognition model for analyzing legal documents.
- 2024-04-08 Supports [Ollama](./docs/ollama.md) for local LLM deployment.
- 2024-04-07 Supports Chinese UI.

## 📌 Latest Updates

Expand Down Expand Up @@ -274,6 +286,7 @@ $ systemctl start nginx

## 📚 Documentation

- [Quickstart](./docs/quickstart.md)
- [FAQ](./docs/faq.md)

## 📜 Roadmap
Expand Down
17 changes: 16 additions & 1 deletion README_ja.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,21 @@

## 💡 RAGFlow とは?

[RAGFlow](https://demo.ragflow.io) は、深い文書理解に基づいたオープンソースの RAG (Retrieval-Augmented Generation) エンジンである。LLM(大規模言語モデル)を組み合わせることで、様々な複雑なフォーマットのデータから根拠のある引用に裏打ちされた、信頼できる質問応答機能を実現し、あらゆる規模のビジネスに適した RAG ワークフローを提供します。
[RAGFlow](https://ragflow.io/) は、深い文書理解に基づいたオープンソースの RAG (Retrieval-Augmented Generation) エンジンである。LLM(大規模言語モデル)を組み合わせることで、様々な複雑なフォーマットのデータから根拠のある引用に裏打ちされた、信頼できる質問応答機能を実現し、あらゆる規模のビジネスに適した RAG ワークフローを提供します。

## 📌 最新情報

- 2024-05-15 OpenAI GPT-4oを統合しました。
- 2024-05-08 LLM DeepSeek-V2を統合しました。
- 2024-04-26 「ファイル管理」機能を追加しました。
- 2024-04-19 会話 API をサポートします ([詳細](./docs/conversation_api.md))。
- 2024-04-16 [BCEmbedding](https://github.com/netease-youdao/BCEmbedding) から埋め込みモデル「bce-embedding-base_v1」を追加します。
- 2024-04-16 [FastEmbed](https://github.com/qdrant/fastembed) は、軽量かつ高速な埋め込み用に設計されています。
- 2024-04-11 ローカル LLM デプロイメント用に [Xinference](./docs/xinference.md) をサポートします。
- 2024-04-10 メソッド「Laws」に新しいレイアウト認識モデルを追加します。
- 2024-04-08 [Ollama](./docs/ollama.md) を使用した大規模モデルのローカライズされたデプロイメントをサポートします。
- 2024-04-07 中国語インターフェースをサポートします。


## 📌 最新情報

Expand Down Expand Up @@ -252,6 +266,7 @@ $ bash ./entrypoint.sh

## 📚 ドキュメンテーション

- [Quickstart](./docs/quickstart.md)
- [FAQ](./docs/faq.md)

## 📜 ロードマップ
Expand Down
15 changes: 14 additions & 1 deletion README_zh.md
Original file line number Diff line number Diff line change
Expand Up @@ -26,7 +26,19 @@

## 💡 RAGFlow 是什么?

[RAGFlow](https://demo.ragflow.io) 是一款基于深度文档理解构建的开源 RAG(Retrieval-Augmented Generation)引擎。RAGFlow 可以为各种规模的企业及个人提供一套精简的 RAG 工作流程,结合大语言模型(LLM)针对用户各类不同的复杂格式数据提供可靠的问答以及有理有据的引用。
[RAGFlow](https://ragflow.io/) 是一款基于深度文档理解构建的开源 RAG(Retrieval-Augmented Generation)引擎。RAGFlow 可以为各种规模的企业及个人提供一套精简的 RAG 工作流程,结合大语言模型(LLM)针对用户各类不同的复杂格式数据提供可靠的问答以及有理有据的引用。

## 📌 近期更新

- 2024-05-15 集成大模型 OpenAI GPT-4o。
- 2024-05-08 集成大模型 DeepSeek。
- 2024-04-26 增添了'文件管理'功能。
- 2024-04-19 支持对话 API ([更多](./docs/conversation_api.md))。
- 2024-04-16 集成嵌入模型 [BCEmbedding](https://github.com/netease-youdao/BCEmbedding) 和 专为轻型和高速嵌入而设计的 [FastEmbed](https://github.com/qdrant/fastembed)
- 2024-04-11 支持用 [Xinference](./docs/xinference.md) 本地化部署大模型。
- 2024-04-10 为‘Laws’版面分析增加了底层模型。
- 2024-04-08 支持用 [Ollama](./docs/ollama.md) 本地化部署大模型。
- 2024-04-07 支持中文界面。

## 📌 近期更新

Expand Down Expand Up @@ -271,6 +283,7 @@ $ systemctl start nginx
```
## 📚 技术文档

- [Quickstart](./docs/quickstart.md)
- [FAQ](./docs/faq.md)

## 📜 路线图
Expand Down
139 changes: 130 additions & 9 deletions api/apps/api_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,19 +13,23 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
import json
import os
import re
from datetime import datetime, timedelta
from flask import request
from flask import request, Response
from flask_login import login_required, current_user

from api.db import FileType, ParserType
from api.db.db_models import APIToken, API4Conversation
from api.db.db_models import APIToken, API4Conversation, Task
from api.db.services import duplicate_name
from api.db.services.api_service import APITokenService, API4ConversationService
from api.db.services.dialog_service import DialogService, chat
from api.db.services.document_service import DocumentService
from api.db.services.file2document_service import File2DocumentService
from api.db.services.file_service import FileService
from api.db.services.knowledgebase_service import KnowledgebaseService
from api.db.services.task_service import queue_tasks, TaskService
from api.db.services.user_service import UserTenantService
from api.settings import RetCode
from api.utils import get_uuid, current_timestamp, datetime_format
Expand All @@ -35,6 +39,9 @@
from api.utils.file_utils import filename_type, thumbnail
from rag.utils.minio_conn import MINIO

from rag.utils.es_conn import ELASTICSEARCH
from rag.nlp import search
from elasticsearch_dsl import Q

def generate_confirmation_token(tenent_id):
serializer = URLSafeTimedSerializer(tenent_id)
Expand Down Expand Up @@ -164,6 +171,7 @@ def completion():
e, conv = API4ConversationService.get_by_id(req["conversation_id"])
if not e:
return get_data_error_result(retmsg="Conversation not found!")
if "quote" not in req: req["quote"] = False

msg = []
for m in req["messages"]:
Expand All @@ -180,13 +188,48 @@ def completion():
return get_data_error_result(retmsg="Dialog not found!")
del req["conversation_id"]
del req["messages"]
ans = chat(dia, msg, **req)

if not conv.reference:
conv.reference = []
conv.reference.append(ans["reference"])
conv.message.append({"role": "assistant", "content": ans["answer"]})
API4ConversationService.append_message(conv.id, conv.to_dict())
return get_json_result(data=ans)
conv.message.append({"role": "assistant", "content": ""})
conv.reference.append({"chunks": [], "doc_aggs": []})

def fillin_conv(ans):
nonlocal conv
if not conv.reference:
conv.reference.append(ans["reference"])
else: conv.reference[-1] = ans["reference"]
conv.message[-1] = {"role": "assistant", "content": ans["answer"]}

def stream():
nonlocal dia, msg, req, conv
try:
for ans in chat(dia, msg, True, **req):
fillin_conv(ans)
yield "data:"+json.dumps({"retcode": 0, "retmsg": "", "data": ans}, ensure_ascii=False) + "\n\n"
API4ConversationService.append_message(conv.id, conv.to_dict())
except Exception as e:
yield "data:" + json.dumps({"retcode": 500, "retmsg": str(e),
"data": {"answer": "**ERROR**: "+str(e), "reference": []}},
ensure_ascii=False) + "\n\n"
yield "data:"+json.dumps({"retcode": 0, "retmsg": "", "data": True}, ensure_ascii=False) + "\n\n"

if req.get("stream", True):
resp = Response(stream(), mimetype="text/event-stream")
resp.headers.add_header("Cache-control", "no-cache")
resp.headers.add_header("Connection", "keep-alive")
resp.headers.add_header("X-Accel-Buffering", "no")
resp.headers.add_header("Content-Type", "text/event-stream; charset=utf-8")
return resp
else:
answer = None
for ans in chat(dia, msg, **req):
answer = ans
fillin_conv(ans)
API4ConversationService.append_message(conv.id, conv.to_dict())
break
return get_json_result(data=answer)

except Exception as e:
return server_error_response(e)

Expand Down Expand Up @@ -233,6 +276,13 @@ def upload():
if file.filename == '':
return get_json_result(
data=False, retmsg='No file selected!', retcode=RetCode.ARGUMENT_ERROR)

root_folder = FileService.get_root_folder(tenant_id)
pf_id = root_folder["id"]
FileService.init_knowledgebase_docs(pf_id, tenant_id)
kb_root_folder = FileService.get_kb_folder(tenant_id)
kb_folder = FileService.new_a_file_from_kb(kb.tenant_id, kb.name, kb_root_folder["id"])

try:
if DocumentService.get_doc_count(kb.tenant_id) >= int(os.environ.get('MAX_FILE_NUM_PER_USER', 8192)):
return get_data_error_result(
Expand Down Expand Up @@ -264,11 +314,82 @@ def upload():
"size": len(blob),
"thumbnail": thumbnail(filename, blob)
}

form_data=request.form
if "parser_id" in form_data.keys():
if request.form.get("parser_id").strip() in list(vars(ParserType).values())[1:-3]:
doc["parser_id"] = request.form.get("parser_id").strip()
if doc["type"] == FileType.VISUAL:
doc["parser_id"] = ParserType.PICTURE.value
if re.search(r"\.(ppt|pptx|pages)$", filename):
doc["parser_id"] = ParserType.PRESENTATION.value
doc = DocumentService.insert(doc)
return get_json_result(data=doc.to_json())

doc_result = DocumentService.insert(doc)
FileService.add_file_from_kb(doc, kb_folder["id"], kb.tenant_id)
except Exception as e:
return server_error_response(e)

if "run" in form_data.keys():
if request.form.get("run").strip() == "1":
try:
info = {"run": 1, "progress": 0}
info["progress_msg"] = ""
info["chunk_num"] = 0
info["token_num"] = 0
DocumentService.update_by_id(doc["id"], info)
# if str(req["run"]) == TaskStatus.CANCEL.value:
tenant_id = DocumentService.get_tenant_id(doc["id"])
if not tenant_id:
return get_data_error_result(retmsg="Tenant not found!")

#e, doc = DocumentService.get_by_id(doc["id"])
TaskService.filter_delete([Task.doc_id == doc["id"]])
e, doc = DocumentService.get_by_id(doc["id"])
doc = doc.to_dict()
doc["tenant_id"] = tenant_id
bucket, name = File2DocumentService.get_minio_address(doc_id=doc["id"])
queue_tasks(doc, bucket, name)
except Exception as e:
return server_error_response(e)

return get_json_result(data=doc_result.to_json())


@manager.route('/list_chunks', methods=['POST'])
# @login_required
def list_chunks():
token = request.headers.get('Authorization').split()[1]
objs = APIToken.query(token=token)
if not objs:
return get_json_result(
data=False, retmsg='Token is not valid!"', retcode=RetCode.AUTHENTICATION_ERROR)

form_data = request.form

try:
if "doc_name" in form_data.keys():
tenant_id = DocumentService.get_tenant_id_by_name(form_data['doc_name'])
q = Q("match", docnm_kwd=form_data['doc_name'])

elif "doc_id" in form_data.keys():
tenant_id = DocumentService.get_tenant_id(form_data['doc_id'])
q = Q("match", doc_id=form_data['doc_id'])
else:
return get_json_result(
data=False,retmsg="Can't find doc_name or doc_id"
)

res_es_search = ELASTICSEARCH.search(q,idxnm=search.index_name(tenant_id),timeout="600s")

res = [{} for _ in range(len(res_es_search['hits']['hits']))]

for index , chunk in enumerate(res_es_search['hits']['hits']):
res[index]['doc_name'] = chunk['_source']['docnm_kwd']
res[index]['content'] = chunk['_source']['content_with_weight']
if 'img_id' in chunk['_source'].keys():
res[index]['img_id'] = chunk['_source']['img_id']

except Exception as e:
return server_error_response(e)

return get_json_result(data=res)
2 changes: 1 addition & 1 deletion api/apps/chunk_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -38,7 +38,7 @@
@manager.route('/list', methods=['POST'])
@login_required
@validate_request("doc_id")
def list():
def list_chunk():
req = request.json
doc_id = req["doc_id"]
page = int(req.get("page", 1))
Expand Down
53 changes: 46 additions & 7 deletions api/apps/conversation_app.py
Original file line number Diff line number Diff line change
Expand Up @@ -13,12 +13,13 @@
# See the License for the specific language governing permissions and
# limitations under the License.
#
from flask import request
from flask import request, Response, jsonify
from flask_login import login_required
from api.db.services.dialog_service import DialogService, ConversationService, chat
from api.utils.api_utils import server_error_response, get_data_error_result, validate_request
from api.utils import get_uuid
from api.utils.api_utils import get_json_result
import json


@manager.route('/set', methods=['POST'])
Expand Down Expand Up @@ -103,9 +104,12 @@ def list_convsersation():

@manager.route('/completion', methods=['POST'])
@login_required
@validate_request("conversation_id", "messages")
#@validate_request("conversation_id", "messages")
def completion():
req = request.json
#req = {"conversation_id": "9aaaca4c11d311efa461fa163e197198", "messages": [
# {"role": "user", "content": "上海有吗?"}
#]}
msg = []
for m in req["messages"]:
if m["role"] == "system":
Expand All @@ -123,13 +127,48 @@ def completion():
return get_data_error_result(retmsg="Dialog not found!")
del req["conversation_id"]
del req["messages"]
ans = chat(dia, msg, **req)

if not conv.reference:
conv.reference = []
conv.reference.append(ans["reference"])
conv.message.append({"role": "assistant", "content": ans["answer"]})
ConversationService.update_by_id(conv.id, conv.to_dict())
return get_json_result(data=ans)
conv.message.append({"role": "assistant", "content": ""})
conv.reference.append({"chunks": [], "doc_aggs": []})

def fillin_conv(ans):
nonlocal conv
if not conv.reference:
conv.reference.append(ans["reference"])
else: conv.reference[-1] = ans["reference"]
conv.message[-1] = {"role": "assistant", "content": ans["answer"]}

def stream():
nonlocal dia, msg, req, conv
try:
for ans in chat(dia, msg, True, **req):
fillin_conv(ans)
yield "data:"+json.dumps({"retcode": 0, "retmsg": "", "data": ans}, ensure_ascii=False) + "\n\n"
ConversationService.update_by_id(conv.id, conv.to_dict())
except Exception as e:
yield "data:" + json.dumps({"retcode": 500, "retmsg": str(e),
"data": {"answer": "**ERROR**: "+str(e), "reference": []}},
ensure_ascii=False) + "\n\n"
yield "data:"+json.dumps({"retcode": 0, "retmsg": "", "data": True}, ensure_ascii=False) + "\n\n"

if req.get("stream", True):
resp = Response(stream(), mimetype="text/event-stream")
resp.headers.add_header("Cache-control", "no-cache")
resp.headers.add_header("Connection", "keep-alive")
resp.headers.add_header("X-Accel-Buffering", "no")
resp.headers.add_header("Content-Type", "text/event-stream; charset=utf-8")
return resp

else:
answer = None
for ans in chat(dia, msg, **req):
answer = ans
fillin_conv(ans)
ConversationService.update_by_id(conv.id, conv.to_dict())
break
return get_json_result(data=answer)
except Exception as e:
return server_error_response(e)

Loading

0 comments on commit a0818bf

Please sign in to comment.