LLM transformer

Configuration

Dependencies

The dependencies for this project are managed by Poetry. To install them, run

poetry install

Some of the dependencies are:

• Pytorch 2.1
• Python 3.10

Docker

A Dockerfile is provided to run the code in a container. To build the image, run

./build_docker_image.sh

The image name is $HOSTNAME/llm-transformer. To run the container, run

./docker.sh python -m llmt.main --help

Hardware

This code was developed and tested on the Nvidia 4090 GPU with 24GB of memory.

Usage

Setup

huggingface-cli login
cp ~/cache/huggingface/token ./data/
poetry install

Token is supposed to be under directory ./data

Dataset

In order to download the dataset, run

./docker.sh python -m llmt.main dataset download

and it will be downloaded under -./data.

Training

In order to train a model, run

./docker.sh python -m llmt.main train

We use the tokenizer from https://huggingface.co/replit/replit-code-v1-3b

Validation and metrics

TODO

Links:

• https://github.com/openai/human-eval-infilling
• https://github.com/nuprl/MultiPL-E

Inference

TODO

TODO

• Implement the test function to evaluate the generated code
• Use sintax trees for the languages to remove the spaces which add no information and may lead to slower learning
• Use sintax trees to change the variable names