Project Name

Description

Mindvault is GPT model with KNN-Memory and Recurrence.

Table of Contents

• Features
• Installation
• Usage
• Contributing

Features

1. KNN : This project uses a FAISS vector database to store keys generated by a single head during forward propagation, the key-value pairs are then store in a numpy.ndarray database to be retrieved later for attention score calculation.

2. Recurrence : This project uses recurrence for keys and values. Each segment is allowed to access the keys and values projected by all other previous segments. The keys and queries projected by the last Decoder layer is added to the recurrence for the next segment to be used.

Installation

To install the project

1. Clone the repository:

   git clone https://github.com/notaryanramani/mindvault.git

2. 2. Install the package.

   pip install -e .
   

   Note - If you encounter an error while installing torch packages, please refer PyTorch documentation to install PyTorch for your system configuration and OS.

Usage

This model is used to train a Decoder only Transformer that uses KNN & Recuurance for attention score from scratch. Please refer to main.py file for sample code.

Contributing

Contributions are welcome, just raise a pull request. Feel free to raise an issue if you encounter an error or bug!