Skip to content

Commit af86bc2

Browse files
Retrieval Augmented Generation with LLM Demo (#16)
- Added a new RAG + prompt + LLM UI (demo). - Added an example config and notebook. - Updated main README with "updates" sub-section. - Updated `run_demo.py` to include all the options to run a demo (UI, UI + service, UI + <user_defined_service>)
1 parent 056c1c6 commit af86bc2

13 files changed

+776
-186
lines changed

README.md

+143-120
Large diffs are not rendered by default.
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,50 @@
1+
components:
2+
- name: Store
3+
params:
4+
host: <index ip>
5+
index: <index name>
6+
port: 80
7+
search_fields: ["title", "content"]
8+
type: ElasticsearchDocumentStore
9+
- name: Retriever
10+
params:
11+
document_store: Store
12+
top_k: 100
13+
type: BM25Retriever
14+
- name: Reranker
15+
params:
16+
batch_size: 32
17+
model_name_or_path: cross-encoder/ms-marco-MiniLM-L-6-v2
18+
top_k: 5
19+
use_gpu: true
20+
type: SentenceTransformersRanker
21+
- name: AParser
22+
type: AnswerParser
23+
- name: LFQA
24+
params:
25+
name: lfqa
26+
prompt_text: "Answer the question using the provided context. Your answer should be in your own words and be no longer than 50 words. \n\n Context: {join(documents)} \n\n Question: {query} \n\n Answer:"
27+
output_parser: AParser
28+
type: PromptTemplate
29+
- name: Prompter
30+
params:
31+
model_name_or_path: MBZUAI/LaMini-Flan-T5-783M
32+
use_gpu: true
33+
model_kwargs:
34+
model_max_length: 2048
35+
torch_dtype: torch.bfloat16
36+
default_prompt_template: LFQA
37+
type: PromptNode
38+
pipelines:
39+
- name: query
40+
nodes:
41+
- inputs:
42+
- Query
43+
name: Retriever
44+
- inputs:
45+
- Retriever
46+
name: Reranker
47+
- inputs:
48+
- Reranker
49+
name: Prompter
50+
version: 1.17.0

demo/README.md

+35-11
Original file line numberDiff line numberDiff line change
@@ -1,26 +1,50 @@
11
# Running Demos
22

3-
To run a demo, use its config name; for example:
3+
To execute a demo, use its configuration name. For instance:
44

55
```sh
66
python run_demo.py -t QA1
77
```
88

9-
The server and UI are are created as subprocesses that run in the background. Use the PIDs to kill them.
9+
The server and UI will be spawned as subprocesses that run in the background. You can use the PIDs (Process IDs) to terminate them when needed.
1010

11-
Use the `--help` flag for a list of available configurations.
11+
To obtain a list of available configurations, utilize the `--help` flag.
1212

1313
## Available Demos
1414

15-
| Name | Comment | Config Name |
16-
|:--------|:------------------------------------------------------------------------------------|:-----------:|
17-
| Q&A | Abstractive Q&A demo using BM25, SBERT reranker and an FiD. | `QA1` |
18-
| Q&A | Abstractive Q&A demo using ColBERT v2 (w/ PLAID index) retriever and an FiD reader. | `QA2` |
19-
| Summary | Summarization using BM25, SBERT reranker and long-T5 reader | `SUM` |
20-
| Image | Abstractive Q&A demo, with an image generation model for the answer. | `QADIFF` |
15+
| Name | Description | Config Name |
16+
|:--------|:-------------------------------------------------------------------------------------|:-----------:|
17+
| Q&A | Abstractive Q&A demo utilizing BM25, SBERT reranker, and FiD model. | `QA1` |
18+
| Q&A | Abstractive Q&A demo using ColBERT v2 (with PLAID index) retriever and FiD reader. | `QA2` |
19+
| Summarization | Summarization demo employing BM25, SBERT reranker, and long-T5 reader. | `SUM` |
20+
| Image | Abstractive Q&A demo with an image generation model for the answer. | `QADIFF` |
21+
| LLM | Retrieval augmented generation with generative LLM model. | `LLM` |
2122

22-
ColBERT demo with a wikipedia index takes about 15 minutes to load up. Also, see remark about GPU usage in the [README](../README.md#plaid-requirements).
23+
Please note that the ColBERT demo with a Wikipedia index may take around 15 minutes to load. Also, make sure to review the [README](../models.md#plaid-requirements) for information regarding GPU usage requirements.
2324

24-
## Demo Screenshot
25+
### Additional Options
26+
27+
If you already have a fastRAG pipeline service running locally and wish to utilize it with one of the provided UI interfaces, you can add the `--only-ui` flag to the demo script:
28+
29+
```sh
30+
python run_demo.py -t LLM --only-ui
31+
```
32+
33+
In case your pipeline service is running on a non-local machine or a different port other than 8000, you can use the `--endpoint` argument to specify the URL:
34+
35+
```sh
36+
python run_demo.py -t LLM --endpoint http://hostname:80
37+
```
38+
39+
To manually run a UI with the `API_ENDPOINT` directed to a fastRAG service, you can execute the following command:
40+
41+
```bash
42+
API_ENDPOINT=http://localhost:8000 \
43+
python -m streamlit run fastrag/ui/webapp.py
44+
```
45+
46+
Make sure to replace `http://localhost:8000` with the appropriate URL of your fastRAG service.
47+
48+
## Screenshot
2549

2650
![alt text](../assets/qa_demo.png)

demo/run_demo.py

+23-11
Original file line numberDiff line numberDiff line change
@@ -8,13 +8,15 @@
88
"QA2": "qa_plaid.yaml",
99
"QADIFF": "qa_diffusion_pipeline.yaml",
1010
"SUMR": "summarization_pipeline.yaml",
11+
"LLM": "rag_generation_with_dynamic_prompt.yaml",
1112
}
1213

1314
SCREENS = {
1415
"QA1": "webapp",
1516
"QA2": "webapp",
1617
"QADIFF": "webapp",
1718
"SUMR": "webapp_summarization",
19+
"LLM": "prompt_llm",
1820
}
1921

2022

@@ -40,27 +42,37 @@ def get_pid(cmd):
4042
choices=list(TASKS.keys()),
4143
help=f"The abbreviated name for the task configuraion. \n {TASKS} \n",
4244
)
45+
parser.add_argument(
46+
"-e", "--endpoint", default="http://localhost:8000", help="pipeline service endpoint"
47+
)
48+
parser.add_argument(
49+
"--only-ui",
50+
action="store_true",
51+
help="launch only the UI interface (without launching a service)",
52+
)
4353

4454
args = parser.parse_args()
4555
path = os.getcwd()
4656

47-
# Create REST server
48-
cmd = f"python -m fastrag.rest_api.application --config={path}/config/TASKCONFIGURATION"
49-
cmd = cmd.replace("TASKCONFIGURATION", TASKS[args.task_config])
50-
run_service(cmd)
57+
s_pid = "NA"
58+
if not args.only_ui:
59+
# Create REST server
60+
cmd = f"python -m fastrag.rest_api.application --config={path}/config/TASKCONFIGURATION"
61+
cmd = cmd.replace("TASKCONFIGURATION", TASKS[args.task_config])
62+
print("Launching fastRAG pipeline service...")
63+
run_service(cmd)
64+
time.sleep(10)
65+
s_pid = get_pid("fastrag.rest_api.application")
5166

5267
# Create UI
53-
os.environ["API_ENDPOINT"] = "http://localhost:8000"
68+
os.environ["API_ENDPOINT"] = f"{args.endpoint}"
5469
cmd = f"python -m streamlit run {path}/fastrag/ui/SCREEN.py"
5570
cmd = cmd.replace("SCREEN", SCREENS[args.task_config])
71+
print("Launching UI...")
72+
time.sleep(3)
5673
run_service(cmd)
57-
58-
# Sleep and wait for initialization, pids
59-
print("Creating services...")
60-
time.sleep(10)
61-
s_pid = get_pid("fastrag.rest_api.application")
6274
u_pid = get_pid("streamlit run")
6375

6476
print("\n")
65-
print(f"Server on localhost:8000/docs PID={s_pid}")
77+
print(f"Server on {args.endpoint}/docs PID={s_pid}")
6678
print(f"UI on localhost:8501 PID={u_pid}")

examples/rag-prompt-hf.ipynb

+164
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,164 @@
1+
{
2+
"cells": [
3+
{
4+
"cell_type": "markdown",
5+
"id": "7af1bfad",
6+
"metadata": {},
7+
"source": [
8+
"# Retrieval Augmented Generation with LLMs"
9+
]
10+
},
11+
{
12+
"cell_type": "markdown",
13+
"id": "be7a0c4a",
14+
"metadata": {},
15+
"source": [
16+
"Define an information source to retrieve from"
17+
]
18+
},
19+
{
20+
"cell_type": "code",
21+
"execution_count": null,
22+
"id": "359f06de",
23+
"metadata": {},
24+
"outputs": [],
25+
"source": [
26+
"from haystack.schema import Document\n",
27+
"from haystack.document_stores import InMemoryDocumentStore\n",
28+
"\n",
29+
"document_store = InMemoryDocumentStore(use_gpu=False, use_bm25=True)\n",
30+
"\n",
31+
"# 4 example documents to index\n",
32+
"examples = [\n",
33+
" \"Lionel Andrés Messi[note 1] (Spanish pronunciation: [ljoˈnel anˈdɾes ˈmesi] (listen); born 24 June 1987), also known as Leo Messi, is an Argentine professional footballer who plays as a forward for Ligue 1 club Paris Saint-Germain and captains the Argentina national team. Widely regarded as one of the greatest players of all time, Messi has won a record seven Ballon d'Or awards,[note 2] a record six European Golden Shoes, and in 2020 was named to the Ballon d'Or Dream Team. Until leaving the club in 2021, he had spent his entire professional career with Barcelona, where he won a club-record 35 trophies, including 10 La Liga titles, seven Copa del Rey titles and four UEFA Champions Leagues. With his country, he won the 2021 Copa América and the 2022 FIFA World Cup. A prolific goalscorer and creative playmaker, Messi holds the records for most goals in La Liga (474), most hat-tricks in La Liga (36) and the UEFA Champions League (8), and most assists in La Liga (192) and the Copa América (17). He has also the most international goals by a South American male (98). Messi has scored over 795 senior career goals for club and country, and has the most goals by a player for a single club (672).\",\n",
34+
" \"Born and raised in central Argentina, Messi relocated to Spain at the age of 13 to join Barcelona, for whom he made his competitive debut aged 17 in October 2004. He established himself as an integral player for the club within the next three years, and in his first uninterrupted season in 2008–09 he helped Barcelona achieve the first treble in Spanish football; that year, aged 22, Messi won his first Ballon d'Or. Three successful seasons followed, with Messi winning four consecutive Ballons d'Or, making him the first player to win the award four times. During the 2011–12 season, he set the La Liga and European records for most goals scored in a single season, while establishing himself as Barcelona's all-time top scorer. The following two seasons, Messi finished second for the Ballon d'Or behind Cristiano Ronaldo (his perceived career rival), before regaining his best form during the 2014–15 campaign, becoming the all-time top scorer in La Liga and leading Barcelona to a historic second treble, after which he was awarded a fifth Ballon d'Or in 2015. Messi assumed captaincy of Barcelona in 2018, and in 2019 he won a record sixth Ballon d'Or. Out of contract, he signed for Paris Saint-Germain in August 2021.\",\n",
35+
" \"An Argentine international, Messi holds the national record for appearances and is also the country's all-time leading goalscorer. At youth level, he won the 2005 FIFA World Youth Championship, finishing the tournament with both the Golden Ball and Golden Shoe, and an Olympic gold medal at the 2008 Summer Olympics. His style of play as a diminutive, left-footed dribbler drew comparisons with his compatriot Diego Maradona, who described Messi as his successor. After his senior debut in August 2005, Messi became the youngest Argentine to play and score in a FIFA World Cup in 2006, and reached the final of the 2007 Copa América, where he was named young player of the tournament. As the squad's captain from August 2011, he led Argentina to three consecutive finals: the 2014 FIFA World Cup, for which he won the Golden Ball, and the 2015 and 2016 Copa América, winning the Golden Ball in the 2015 edition. After announcing his international retirement in 2016, he reversed his decision and led his country to qualification for the 2018 FIFA World Cup, a third-place finish at the 2019 Copa América, and victory in the 2021 Copa América, while winning the Golden Ball and Golden Boot for the latter. This achievement would see him receive a record seventh Ballon d'Or in 2021. In 2022, he captained his country to win the 2022 FIFA World Cup, for which he won the Golden Ball for a record second time, and broke the record for most appearances in World Cup tournaments with 26 matches played.\",\n",
36+
" \"Messi has endorsed sportswear company Adidas since 2006. According to France Football, he was the world's highest-paid footballer for five years out of six between 2009 and 2014, and was ranked the world's highest-paid athlete by Forbes in 2019 and 2022. Messi was among Time's 100 most influential people in the world in 2011 and 2012. In February 2020, he was awarded the Laureus World Sportsman of the Year, thus becoming the first footballer and the first team sport athlete to win the award. Later that year, Messi became the second footballer and second team-sport athlete to surpass $1 billion in career earnings.\",\n",
37+
" \n",
38+
"]\n",
39+
"\n",
40+
"documents = []\n",
41+
"for i, d in enumerate(examples):\n",
42+
" documents.append(Document(content=d, id=i))\n",
43+
"\n",
44+
"document_store.write_documents(documents)"
45+
]
46+
},
47+
{
48+
"attachments": {},
49+
"cell_type": "markdown",
50+
"id": "7e653e5f",
51+
"metadata": {},
52+
"source": [
53+
"Define the prompt template. `{query}` will be replaced with the user's query and `{documents}` with the retrieved documents fetched from the index.\n",
54+
"\n",
55+
"We define a `PromptModel` that automatically uses a Huggingface model interface given by `model_name_or_path`.\n",
56+
"\n",
57+
"Use `{query}` for injecting the original query text into the prompt and `{documents}` to inject the documents fetched by the retriever (can be used with smaller manipulation functions such as `join()` to concatenate the documents)."
58+
]
59+
},
60+
{
61+
"cell_type": "code",
62+
"execution_count": null,
63+
"id": "6443e7a3",
64+
"metadata": {},
65+
"outputs": [],
66+
"source": [
67+
"import torch\n",
68+
"from haystack.nodes import PromptNode, PromptTemplate\n",
69+
"from haystack.nodes import BM25Retriever, SentenceTransformersRanker\n",
70+
"\n",
71+
"retriever = BM25Retriever(document_store=document_store, top_k=100)\n",
72+
"reranker = SentenceTransformersRanker(model_name_or_path=\"cross-encoder/ms-marco-MiniLM-L-12-v2\", top_k=1)\n",
73+
"\n",
74+
"\n",
75+
"lfqa_prompt = PromptTemplate(name=\"lfqa\",\n",
76+
" prompt_text=\"Answer the question using the provided context. Your answer should be in your own words and be no longer than 50 words. \\n\\n Context: {join(documents)} \\n\\n Question: {query} \\n\\n Answer:\",\n",
77+
" output_parser={\"type\": \"AnswerParser\"}) \n",
78+
"prompt = PromptNode(model_name_or_path=\"MBZUAI/LaMini-Flan-T5-783M\", default_prompt_template=lfqa_prompt,\n",
79+
" model_kwargs={\"model_max_length\": 2048, \"torch_dtype\": torch.bfloat16},)"
80+
]
81+
},
82+
{
83+
"cell_type": "markdown",
84+
"id": "04408d03",
85+
"metadata": {},
86+
"source": [
87+
"Defining the pipeline"
88+
]
89+
},
90+
{
91+
"cell_type": "code",
92+
"execution_count": 3,
93+
"id": "4652b226",
94+
"metadata": {},
95+
"outputs": [],
96+
"source": [
97+
"from haystack import Pipeline\n",
98+
"p = Pipeline()\n",
99+
"p.add_node(component=retriever, name=\"Retriever\", inputs=[\"Query\"])\n",
100+
"p.add_node(component=reranker, name=\"Reranker\", inputs=[\"Retriever\"])\n",
101+
"p.add_node(component=prompt, name=\"prompt_node\", inputs=[\"Reranker\"])"
102+
]
103+
},
104+
{
105+
"cell_type": "markdown",
106+
"id": "0f842709",
107+
"metadata": {},
108+
"source": [
109+
"Run a query through the pipeline and print the generated answer"
110+
]
111+
},
112+
{
113+
"cell_type": "code",
114+
"execution_count": 4,
115+
"id": "3dd989ac",
116+
"metadata": {},
117+
"outputs": [
118+
{
119+
"data": {
120+
"text/plain": [
121+
"'Messi has won a club-record 35 trophies, including 10 La Liga titles, seven Copa del Rey titles, and four UEFA Champions Leagues. He has also won the 2021 Copa América and the 2022 FIFA World Cup.'"
122+
]
123+
},
124+
"execution_count": 4,
125+
"metadata": {},
126+
"output_type": "execute_result"
127+
}
128+
],
129+
"source": [
130+
"a = p.run(\"What trophies does Messi has?\", debug=True)\n",
131+
"a['answers'][0].answer"
132+
]
133+
},
134+
{
135+
"cell_type": "code",
136+
"execution_count": null,
137+
"id": "65963ad0-ac72-4073-ad8d-cf3d459ea5d5",
138+
"metadata": {},
139+
"outputs": [],
140+
"source": []
141+
}
142+
],
143+
"metadata": {
144+
"kernelspec": {
145+
"display_name": "Python 3 (ipykernel)",
146+
"language": "python",
147+
"name": "python3"
148+
},
149+
"language_info": {
150+
"codemirror_mode": {
151+
"name": "ipython",
152+
"version": 3
153+
},
154+
"file_extension": ".py",
155+
"mimetype": "text/x-python",
156+
"name": "python",
157+
"nbconvert_exporter": "python",
158+
"pygments_lexer": "ipython3",
159+
"version": "3.10.11"
160+
}
161+
},
162+
"nbformat": 4,
163+
"nbformat_minor": 5
164+
}

fastrag/__init__.py

+1-1
Original file line numberDiff line numberDiff line change
@@ -4,7 +4,7 @@
44
from fastrag import image_generators, kg_creators, rankers, readers, retrievers, stores
55
from fastrag.utils import add_timing_to_pipeline
66

7-
__version__ = "1.1.0"
7+
__version__ = "1.2.0"
88

99

1010
def load_pipeline(config_path: str) -> Pipeline:

0 commit comments

Comments
 (0)