A curated list of awesome publications and researchers on prompting framework updated and maintained by The Intelligent System Security (IS2).
Large language models (LLMs) have made significant advancements in both academia and industry, bringing about a fundamental engineering paradigm shift in many areas. While LLMs are powerful, it is also crucial to best use their power where "prompt"🔧 plays a core role. However, the booming LLMs themselves, including excellent APIs like ChatGPT, have several inherent limitations: 1) temporal lag of training data, and 2) the lack of physical capabilities to perform external actions.
Recently, we have observed the trend of utilizing prompt-based tools to better utilize the power of LLMs for downstream tasks, but a lack of systematic literature and standardized terminology, partly due to the rapid evolution of this field. Therefore, we survey related prompting tools and promote the concept of the 🌟 "Prompting Framework" (PF) 🌟, i.e. the framework for managing, simplifying, and facilitating interaction with large language models. We define the lifecycle of the PF as a hierarchical structure, from bottom to top, namely: Data Level, Base Level, Execute Level, and Service Level.
We also systematically depict the overall landscape of the emerging PF field and discuss potential future research and challenges. To continuously track the developments in this area, we maintain this repo, which can be a useful resource sharing platform for both academic and industry in this field.
The Prompting Framework is a systematic architecture designed for managing, simplifying, and facilitating interactions with LLMs. The framework consists of two primary components: Core PF_Core and Extend PF_Extend, where Core represents the essential functionality required for the basic operation of the framework and Extend provides additional features to enhance its capabilities. Specifically, we define the framework PF as: PF = (PF_Core, PF_Extend)
with the following properties:
- PF_Core = (M, T, I, O), i.e., the Core component must include all four modules: M (Model Collection), T (Task Management Module), I (Input Management Module), and O (Output Management Module).
- PF_Core ⊂ PF, i.e., a PF must include at least the complete Core component PF_Core.
Core Component. The Core component is the essential part of the PF, and any implementation that satisfies the Core requirements is considered a valid instance of PF. The Core is a quadruple, which consists of four key modules, which are formally defined as: PF_Core = (M, T, I, O) where each module is defined as follows:
- Model Collection (M): This module defines the set of language models available for interaction within the framework.
- Task Management Module (T): This module manages and organizes tasks that the system aims to complete through interactions with the language models, and is also responsible for coordinating interactions between modules and controlling the flow of information.
- Input Management Module (I): This module is responsible for preprocessing and structuring the input data to ensure it is correctly formatted for efficient processing by the language models.
- Output Management Module (O): This module handles the processing, formatting, and presentation of the outputs generated by the language models.
Extend Component. The Extend component is optional and includes additional modules that extend the framework's functionality. We define the Extend component as: PF_Extend = (C, A, D, P), where each module is defined as follows:
- Context and Memory Management Module (C): This module manages context and memory for interactions with the language models, allowing the system to track historical information and maintain coherent conversations.
- External Tools and Services Integration Interface (A): This module facilitates the integration of external tools or services, providing additional functionalities such as database interactions, API calls, or external applications.
- Data Processing Module (D): This module handles further processing of input and output data, such as data cleaning, transformation, or enhancement, to support more complex use cases.
- Prompt Collection (P): This module contains various types of prompts and associated techniques used to interact with the language models. The prompts may include fixed templates, dynamically generated prompts, or customized ones based on specific tasks.
In conclusion, the Prompting Framework (PF) provides a modular and efficient architecture for interacting with LLMs. It separates core functionalities from advanced features, ensuring flexibility and adaptability. The Core component, which includes model collection, task management, and input/output modules, forms the foundation for efficient operations. The Extend component adds capabilities such as context management, external tool integration, data processing, and dynamic prompt generation, broadening its applicability. This structure overcomes the limitations of traditional frameworks, offering a scalable, context-aware solution for optimized LLM performance.
📄 Paper Link: Prompting Frameworks for Large Language Models: A Survey
Looking forward to your attention and lively discussion！
Sincerely welcome:
🙋♂️ Share good prompting frameworks with us!!!
🙋♀️ Share your experience of using different prompting frameworks and your urgent needs and services!!!
🙋 Brainstorm the future shape of prompting frameworks!!!
and more ...
📮 Contact Details: [email protected]
🎬 【2025.01.14】Latest version of "Prompting Frameworks for Large Language Models: A Survey" published!
🎬 【2023.11.21】First version of "Prompting Frameworks for Large Language Models: A Survey" published on arXiv!
- Paper Structure
- Related Resource
- State-of-the-art Prompting Frameworks
- Comparative Analysis of Prompting Frameworks
- Key Components of Prompting Frameworks
Section 2 presents background knowledge of the Prompting Framework, including the characteristics of LLMs and the necessity of the Propmpting Framework. Section 3 describes the investigation, including the methodologies and results. Section 4 reviews prompt-related key conceptions, including prompt, prompting, prompt engineering, and prompt learning. Section 5 provides the systematic definitions and taxonomy of Prompting Frameworks. Section 6 presents the comparison and challenges across various dimensions of various Prompting Frameworks. Section 7 reviews prompt-related works and toolkits outside the scope of the Prompting Framework, but can synergistically support and enhance interactions with LLMs in conjunction with prompting frameworks. Section 8 presents the future directions of the Prompting Framework and the potential developments and challenges in more domains.
Fingers crossed these awesome resources sprinkle some extra magic on your tasks too!
| 🐾 Tool | 🌱 Description |
|---|---|
| Trending Papers | A useful tool to help us researchers find and explore papers worth reading first. |
| Connected Papers | Explore academic papers in a visual graph. |
Taking into consideration the technical features, design objectives, and application scenarios, the current prompting framework can be broadly covered by three types: The Shell of LLMs (LLM-SH), Language for Interaction with LLMs (LLM-LNG), and Output Restrictors of LLMs (LLM-RSTR). The rationale behind designing the prompting framework is to facilitate the interaction between LLMs and the external world, and different types of prompting frameworks manifest this enhancement effect from different perspectives.
LLM-SH functions are much like a shell or interface layer in computer systems, emphasizing interaction with LLMs by facilitating their engagement with highly capable third parties, thereby enabling stronger interaction between LLMs, users, and external models.
-
🤖 Universal LLM-SH
🦊 Prompting Framework 🧚♂️ Subcategory 🔍 Introduction Haystack Universal LLM-SH LLM orchestration framework to build customizable, production-ready LLM applications. Connect components (models, vector DBs, file converters) to pipelines or agents that can interact with your data. With advanced retrieval methods, it's best suited for building RAG, question answering, semantic search or conversational agent chatbots. Semantic Kernel Universal LLM-SH Integrate cutting-edge LLM technology quickly and easily into your apps. Langchain Universal LLM-SH Building applications with LLMs through composability. Griptape Universal LLM-SH Modular Python framework for AI agents and workflows with chain-of-thought reasoning, tools, and memory. Griptape is an enterprise grade alternative to LangChain. PromptFlow Universal LLM-SH Build high-quality LLM apps - from prototyping, testing to production deployment and monitoring. LLM-chain Universal LLM-SH A powerful rust crate for building chains in large language models allowing you to summarise text and complete complex tasks. LinGoose Universal LLM-SH LinGoose it's a Go framework for developing LLMs-based application using pipelines. LLMStack Universal LLM-SH No-code platform to build generative AI apps, chatbots and agents with your data. OpenDAN Universal LLM-SH OpenDAN is an open source Personal AI OS , which consolidates various AI modules in one place for your personal use. Hyv Universal LLM-SH Chaining AI & API agents to streamline software development and achieve goals collaboratively. -
🤖 Domain-Specific LLM-SH
🦊 Prompting Framework 🧚♂️ Subcategory 🔍 Introduction LlamaIndex Domain-Specific LLM-SH LlamaIndex (formerly GPT Index) is a data framework for your LLM applications. Embedchain Domain-Specific LLM-SH Data platform for LLMs - Load, index, retrieve and sync any unstructured data. AgentVerse Domain-Specific LLM-SH AgentVerse is designed to facilitate the deployment of multiple LLM-based agents in various applications, which primarily provides two frameworks: task-solving and simulation. SuperAGI Domain-Specific LLM-SH A dev-first open source autonomous AI agent framework. Enabling developers to build, manage & run useful autonomous agents quickly and reliably. Txtai Domain-Specific LLM-SH All-in-one open-source embeddings database for semantic search, LLM orchestration and language model workflows. AutoChain Domain-Specific LLM-SH Build lightweight, extensible, and testable LLM Agents. TermGPT Domain-Specific LLM-SH Giving LLMs like GPT-4 the ability to plan and execute terminal commands. Botpress Domain-Specific LLM-SH The open-source hub to build & deploy GPT/LLM Agents.
LLM-LNG, on the other hand, is designed to create a language (programming or pseudo-language) for interaction with LLMs, focusing on providing users with a more concise and compact interaction channel.
-
🤖 Programming LLM-LNG
🦊 Prompting Framework 🧚♂️ Subcategory 🔍 Introduction LMQL Programming LLM-LNG A language for constraint-guided and efficient LLM programming. -
🤖 Pseudocode LLM-LNG
🦊 Prompting Framework 🧚♂️ Subcategory 🔍 Introduction PromptLang Pseudocode LLM-LNG A Prompt based programming language for prompts and AI interactions. Simple and human-readable syntax for easy integration with APIs and data. SudoLang Pseudocode LLM-LNG SudoLang is a programming language designed to collaborate with AI language models including ChatGPT, Bing Chat, Anthropic Claude, and Google Bard. It is designed to be easy to learn and use. It is also very expressive and powerful. gpt-jargon Pseudocode LLM-LNG Jargon is a natural language programming language specified and executed by LLMs like GPT-4.
LLM-RSTR, meanwhile, achieves controlled generation by emphasizing interactions with LLMs that are of higher quality and better aligned with requirements.
-
🤖 Content LLM-RSTR
🦊 Prompting Framework 🧚♂️ Subcategory 🔍 Introduction NeMo-Guardrails Content LLM-RSTR NeMo Guardrails is an open-source toolkit for easily adding programmable guardrails to LLM-based conversational systems. Guardrails Content LLM-RSTR Guardrails is a Python package that lets a user add structure, type and quality guarantees to the outputs of large language models (LLMs). -
🤖 Structure LLM-RSTR
🦊 Prompting Framework 🧚♂️ Subcategory 🔍 Introduction Guidance Structure LLM-RSTR guidance is a programming paradigm that offers superior control and efficiency compared to conventional prompting and chaining. It allows users to constrain generation (e.g. with regex and CFGs) as well as to interleave control (conditional, loops) and generation seamlessly. Promptify Structure LLM-RSTR Prompt Engineering, Solve NLP Problems with LLM's & Easily generate different NLP Task prompts for popular generative models like GPT, PaLM, and more with Promptify. ReLLM Structure LLM-RSTR Exact structure out of any language model completion. TypeChat Structure LLM-RSTR TypeChat is a library that makes it easy to build natural language interfaces using types.
Comparison of state-of-the-art prompting frameworks follows three macro dimensions: 1) compatibility, 2) capabilities and features, 3) documentation and support. The more detailed dimensions are shown below.
