Demo of data-prep-kit for RAG

[Bytes-Explorer]

We want to build a demo using data-prep-kit to showcase RAG functionalities.

Here is the plan for the same:

Input Docs: Enterprise documents (10-15). Good to find documents that have some overlap.

Modules:

1. PDF Extraction Exiting PR Add pdf2parquet transform #416 Owner: @dolfim-ibm
2. [New] Creating chunks from extracted PDF Issue: [Feature] Capability to chunk text for RAG systems #447 Owner: @dolfim-ibm
3. [New] Create embeddings from the given chunks [Feature] Create vector embeddings  #448, Owner: @dolfim-ibm
4. [New] Store the embeddings in vector DB , [Feature] Create a demo notebook for RAG #451 Owner: Sujee
5. [New] Create a notebook with Q&A facility on PDF documents that were ingested. [Feature] Create a demo notebook for RAG #451 Owner: Sujee

Proposed timelines doing backward calculations
Demo Date: Aug 8

Internal version 1 readiness date: 2 Aug

Refine demo, technical correctness: 3 Aug - 6 Aug

PR Reviewer
David Wood

Program Management
Shahrokh Daijavad