LLM: llama-cpp

[kreneskyp]

Description

Adds initial support for Llama-cpp LLM for running local models. This enables the model to be used but streaming and some other things don't work exactly right yet.

Setup

1. download models:
   Tested with GGUF based models from huggingface

   • https://huggingface.co/TheBloke/CodeLlama-7B-GGUF
   • https://huggingface.co/TheBloke/CodeLlama-13B-Python-GGUF

2. save model to <ix root>/llama/<model>

3. create LLM + chain

   set model_path to /var/app/ix/llama/<model>

   

Changes

Adds LLAMA_CPP_LLM

How Tested

• manual testing

TODOs

• Streaming isn't working with LLAMA_CPP_LLM:

  • IxHandler isn't receiving all kwargs the model is initialized with so can't tell if streaming was enabled. This is a potential blocker. LLAMA_CPP appears to be intentionally filtering these out from invocation params
  • IxHandler needs to be updated to start streaming for LLMs (only supported for chat models right now)
  • Might be necessary to implement streaming using the officially blessed method of iterating over chain.astream() if workaround can't be found.

• Docker image isn't setup for GPU acceleration. I made a short attempt at adding libraries to compile GPU support with ENV LLAMA_CUBLAS=1 but the required libraries weren't installed in python:3.11 docker image. Wasn't readily apparent how to install the library. May require switching to a different base image with better support.