Instructions for Designing Your Experiments and Creating a Motivation Example

[pooyanjamshidi]

Specific Task:

You are developing a chatbot for the University of South Carolina, using the RAG framework to retrieve campus-specific information. Your challenge is ensuring that the retrieval and generation steps are efficient and relevant.

Experiment Design:

• Data Retrieval: Start by setting up your web scraping tool to gather data from USC-affiliated websites. Your initial experiment should focus on how effectively your system retrieves relevant documents.
• LLM Evaluation: Test different LLMs for generating answers. Compare their performance based on retrieval accuracy, response relevance, and latency.

Motivation Example:

Present a plot comparing retrieval times and accuracy of answers generated by different LLMs in response to campus-related queries (e.g., "When is the next basketball game?" or "What courses are offered in Spring?"). This will help demonstrate the effectiveness of your system.

Evaluation Focus:

• Retrieval metrics like Mean Reciprocal Rank (MRR), Precision@k, and Recall@k for the RAG system.
• LLM quality using metrics like latency, BLEU, ROUGE, and semantic similarity to validate the accuracy of answers.