Improve installation process

[abetlen]

Open to suggestions / assistance on how to make installation easier and less error prone.

One thought is to add better platform detection to the cmakelists and provide better docs / links if required environment variables aren't set / libraries can't be found.

[ElliottDyson]

GPU detection to decide on what llama.cpp build to use? For example, if Arc is detected, but no Nvidia nor AMD card, then use the build that actively supports it.

As for the current process it has to be built anew each time, which is particularly a pain for those that struggle with this process with the various specific compilers that have to be used for OneAPI.

[vriesdemichael]

Wouldn't llama.cpp be the logical place to implement these improvements?

One thing improvement that would logically fall in the responsibility of llama-cpp-python is using the prebuilt llama.cpp when using windows, but other than that it would just be another layer of cmake

[kristaller486]

Prebuilt wheels with GPU support for all platforms (on Github or PyPI). According to my observations, installing llama-cpp-python with GPU support is the most popular problem when installing llama-cpp-python, prebuilt packages should fix it.

[abetlen]

Prebuilt wheels with GPU support for all platforms (on Github or PyPI). According to my observations, installing llama-cpp-python with GPU support is the most popular problem when installing llama-cpp-python, prebuilt packages should fix it.

I'm partial to this, PyPI is a little annoying because we would need different package names for each but if we did it using seperate indexes (similar to pytorch) this should work. Ideally this would be done via seperate index URLs for metal, CUDA, etc. Maybe can be done with a GitHub pages run on each release.

Update : Started working on this and it's very much do-able, the process is straightforward and much of the hard work to figure out how to build these wheels has thankfully been done by @jllllll and @oobabooga

1. On new releases we can build wheels for each target (trying to keep number to a minimum).
2. Generate an index html file for each target
3. Deploy to github pages

Basic example of this in https://github.com/abetlen/github-pages-pypi-index

As an example, to install the latest llama-cpp-python version:

pip install llama-cpp-python --extra-index-url https://abetlen.github.io/github-pages-pypi-index/whl/cpu

In the future this will likely be found at https://abetlen.github.io/llama-cpp-python/whl/cpu

[DvitryG]

will the installation without wheels be supported? I just tried to update the package after a long break and got an error: Getting requirements to build wheel did not run successfully.

This is the first time I've come across wheels, so I want to ask if performance will deteriorate from using pre-build libraries? I don't have the most powerful computer, so it's important for me to make the most of it.

[abetlen]

@DvitryG yes source installation will always be the default (pip install llama-cpp-python etc) and it should offer the most control / performance.

[waheedi]

Thank you for making the effort to create these bindings, it would have been a nightmare ;D

One important improvement, specifically if building from a cloned source repo, is to keep the newly built wheel in place, maybe somewhere like dist/llama_cpp_python-0.2.76-cp310-cp310-linux_x86_64.whl

Currently there is no wheel you can grab in your hand but rather its directly installed into packages

It seems the setup.py way is going obsolete.. But it still can be used independently.

I just made it to work with llama for now :)

import os
import subprocess
from setuptools import setup, find_packages
from setuptools.command.build_ext import build_ext

class CMakeBuild(build_ext):
    def run(self):
        # Ensure CMake is installed
        try:
            subprocess.check_call(['cmake', '--version'])
        except OSError:
            raise RuntimeError("CMake must be installed to build the following extensions: " +
                               ", ".join(e.name for e in self.extensions))

        # Directory where the current setup.py is located
        source_dir = os.path.abspath(os.path.dirname(__file__))
        build_temp = self.build_temp

        # Create the build directory if it doesn't exist
        if not os.path.exists(build_temp):
            os.makedirs(build_temp)

        # Define the CMake arguments
        cmake_args = [
            '-D', 'LLAMA_HIPBLAS=ON',
            '-D', 'CMAKE_C_COMPILER=/opt/rocm/llvm/bin/clang',
            '-D', 'CMAKE_CXX_COMPILER=/opt/rocm/llvm/bin/clang++',
            '-D', 'CMAKE_PREFIX_PATH=/opt/rocm',
            source_dir  # Add the source directory
        ]

        # Change to the build directory and run CMake
        subprocess.check_call(['cmake'] + cmake_args, cwd=build_temp)
        subprocess.check_call(['make', '-j8'], cwd=build_temp)

        # Ensure the shared library is moved to the expected location

#build/temp.linux-x86_64-cpython-310/libllama.so

        build_lib = os.path.abspath(self.build_lib)
        lib_path = os.path.join(build_temp, 'vendor/llama.cpp/libllama.so')
        target_path = os.path.join(source_dir, 'llama_cpp', 'libllama.so')
        self.copy_file(lib_path, target_path)

setup(
    name='llama_cpp_python',
    version='0.2.76',  # Adjust the version as necessary
    author='abetlen',
    description='Python bindings for llama.cpp',
    long_description=open('README.md').read(),
    long_description_content_type='text/markdown',
    url='https://github.com/abetlen/llama-cpp-python',
    packages=find_packages(),
    include_package_data=True,
    install_requires=[
        'numpy',
    ],
    package_data={
        'llama_cpp': ['libllama.so'],
    },
    classifiers=[
        'Programming Language :: Python :: 3',
        'License :: OSI Approved :: MIT License',
        'Operating System :: OS Independent',
    ],
    python_requires='>=3.6',
    cmdclass={
        'build_ext': CMakeBuild,
    },
)