[Feature Request] plugins (GPTs like action based on function call)

[lloydzhou]

🥰 Feature Description

1. config plugin, like GPTs action (openapi schema + authentication).
2. wrap API to javascript function, then using function-calling`
   

🧐 Proposed Solution

1. manager plugins
   
   
2. using plugin in chat session
   

📝 Additional Information

#5331

[lloydzhou]

#165 #815 #847

[lloydzhou]

WolframAlpha plugin

1. config appid in Authentication
   

2. input wolframalpha llm-api schema:

{
  "openapi": "3.1.0",
  "info": {
    "title": "WolframAlpha",
    "description": "A wrapper around Wolfram Alpha. Useful for when you need to answer questions about Math, Science, Technology, Culture, Society and Everyday Life. Input should be a search query. If the result contains an image link, use the markdown syntax to return the image.",
    "version": "v1.0.0"
  },
  "servers": [
    {
      "url": "https://www.wolframalpha.com"
    }
  ],
  "paths": {
    "/api/v1/llm-api": {
      "get": {
        "operationId": "Calculate",
        "description": "A wrapper around Wolfram Alpha. Useful for when you need to answer questions about Math, Science, Technology, Culture, Society and Everyday Life. Input should be a search query. If the result contains an image link, use the markdown syntax to return the image.",
        "deprecated": false,
        "parameters": [
          {
            "name": "input",
            "in": "query",
            "required": true,
            "description": "questions about Math, Science, Technology, Culture, Society and Everyday Life",
            "schema": {
              "type": "string"
            }
          }
        ]
      }
    }
  },
  "components": {
    "schemas": {}
  }
}

3. then using plugin when chat to api:
   

[lloydzhou]

arxiv search plugin

{
  "openapi": "3.1.0",
  "info": {
    "title": "arxiv search",
    "description": "Run Arxiv search and get the article information.",
    "version": "v1.0.0"
  },
  "servers": [
    {
      "url": "https://export.arxiv.org"
    }
  ],
  "paths": {
    "/api/query": {
      "get": {
        "operationId": "ArxivSearch",
        "description": "Run Arxiv search and get the article information.",
        "deprecated": false,
        "parameters": [
          {
            "name": "search_query",
            "in": "query",
            "required": true,
            "description": "same as the search_query parameter rules of the arxiv API.",
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "sortBy",
            "in": "query",
            "description": "can be `relevance`, `lastUpdatedDate`, `submittedDate`.",
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "sortOrder",
            "in": "query",
            "description": "can be either `ascending` or `descending`.",
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "start",
            "in": "query",
            "description": "the index of the first returned result.",
            "schema": {
              "type": "number"
            }
          },
          {
            "name": "max_results",
            "in": "query",
            "description": "the number of results returned by the query.",
            "schema": {
              "type": "number"
            }
          }
        ]
      }
    }
  },
  "components": {
    "schemas": {}
  }
}

[lloydzhou]

DuckDuckGoLiteSearch

{
  "openapi": "3.1.0",
  "info": {
    "title": "duckduckgo lite",
    "description": "a search engine. useful for when you need to answer questions about current events. input should be a search query.",
    "version": "v1.0.0"
  },
  "servers": [
    {
      "url": "https://lite.duckduckgo.com"
    }
  ],
  "paths": {
    "/lite/": {
      "post": {
        "operationId": "DuckDuckGoLiteSearch",
        "description": "a search engine. useful for when you need to answer questions about current events. input should be a search query.",
        "deprecated": false,
        "parameters": [
          {
            "name": "q",
            "in": "query",
            "required": true,
            "description": "keywords for query.",
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "s",
            "in": "query",
            "description": "can be `0`",
            "schema": {
              "type": "number"
            }
          },
          {
            "name": "o",
            "in": "query",
            "description": "can be `json`",
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "api",
            "in": "query",
            "description": "can be `d.js`",
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "kl",
            "in": "query",
            "description": "wt-wt, us-en, uk-en, ru-ru, etc. Defaults to `wt-wt`.",
            "schema": {
              "type": "string"
            }
          },
          {
            "name": "bing_market",
            "in": "query",
            "description": "wt-wt, us-en, uk-en, ru-ru, etc. Defaults to `wt-wt`.",
            "schema": {
              "type": "string"
            }
          }
        ]
      }
    }
  },
  "components": {
    "schemas": {}
  }
}

[lloydzhou]

gapier

1. config api key
2. load from url: https://a.gapier.com/free
3. using in chat
   

[lloydzhou]

Webpilot

1. config api key
2. load from url: https://gpts.webpilot.ai/gpts-openapi.yaml

[lloydzhou]

FastGPT Dataset

1. config plugin with schema, and apikey

{
  "openapi": "3.1.0",
  "info": {
    "title": "FastGPT",
    "description": "FastGPT 是一个基于 LLM 大语言模型的知识库问答系统，提供开箱即用的数据处理、模型调用等能力。同时可以通过 Flow 可视化进行工作流编排，从而实现复杂的问答场景！",
    "version": "v1.0.0"
  },
  "servers": [
    {
      "url": "https://api.fastgpt.in"
    }
  ],
  "paths": {
    "/api/core/dataset/list": {
      "post": {
        "operationId": "GetDatasetList",
        "description": "Get Dataset list with `id`, `name` and other info",
        "deprecated": false,
        "requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "type": "object",
                "properties": {
                  "parentId": {
                    "type": "string"
                  }
                }
              }
            }
          }
        }
      }
    },
    "/api/core/dataset/searchTest": {
      "post": {
        "operationId": "SearchFromDatasetById",
        "description": "search chunks from dataset by `datesetId`",
        "deprecated": false,"requestBody": {
          "required": true,
          "content": {
            "application/json": {
              "schema": {
                "type": "object",
                "required": ["datasetId", "text"],
                "properties": {
                  "datasetId": {
                    "type": "string",
                    "description": "id for dataset, can query from `GetDatasetList`"
                  },
                  "text": {
                    "type": "string",
                    "description": "keyword to query from dataset"
                  },
                  "limit": {
                    "type": "number",
                    "description": "limit for response data."
                  },
                  "searchMode": {
                    "type": "string",
                    "description": "can be `embedding`"
                  },
                  "usingReRank": {
                    "type": "boolean",
                    "description": "can be `false`"
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  "components": {
    "schemas": {}
  }
}

2. add prompt to chat session:

please get `datasetId` which name is "NextChat README", and query context from this dataset, and then answer user's question.

3. and then ask question, will call GetDatasetList first to get datasetId, and then call SearchFromDatasetById get context, and send context to LLM.
   

[lloydzhou]

awesome plugins

will move plugin demos to another repo from this issue comment.

https://github.com/ChatGPTNextWeb/NextChat-Awesome-Plugins

[Why-SVIP]

您好，可以增加PDF分析的功能吗？

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically.

---

Hello, can you add PDF analysis function?

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically.

---

Hello, can you add PDF analysis function?

Now supported, please check the NextChat-Awesome-Plugins repository

[mountainguan]

plugins功能是不是应该也可以支持gemini的function call，因为gemini有免费额度，使用插件更加畅快

[chidaolatiao]

@lloydzhou 您好，请问可以增加Consensus这一插件的适配吗？如果可以的话希望可以把Scholar GPT一起适配上，感谢。

[Issues-translate-bot]

Bot detected the issue body's language is not English, translate it automatically.

---

@lloydzhou Hello, could you please increase the adaptation of the Consensus plug-in? If possible, I hope that Scholar GPT can be adapted together, thank you.