evaluating-all-minilm-l6-v2-with-llama-cpp.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"name":"python","version":"3.10.14","mimetype":"text/x-python","codemirror_mode":{"name":"ipython","version":3},"pygments_lexer":"ipython3","nbconvert_exporter":"python","file_extension":".py"},"kaggle":{"accelerator":"gpu","dataSources":[{"sourceId":198397091,"sourceType":"kernelVersion"},{"sourceId":120929,"sourceType":"modelInstanceVersion","modelInstanceId":101726,"modelId":125945},{"sourceId":120930,"sourceType":"modelInstanceVersion","modelInstanceId":101727,"modelId":125945}],"dockerImageVersionId":30776,"isInternetEnabled":true,"language":"python","sourceType":"notebook","isGpuEnabled":true}},"nbformat_minor":4,"nbformat":4,"cells":[{"cell_type":"markdown","source":"# Introduction\nThis note evaluate all-MiniLM-L6-v2 with llama.cpp\n\n# Use pre-complie llama.cpp","metadata":{}},{"cell_type":"code","source":"!cp -r /kaggle/input/llama-cpp-binary-compiled-with-gpu/llama.cpp /kaggle/working/","metadata":{"execution":{"iopub.status.busy":"2024-09-28T05:53:49.820245Z","iopub.execute_input":"2024-09-28T05:53:49.821168Z","iopub.status.idle":"2024-09-28T06:15:43.987821Z","shell.execute_reply.started":"2024-09-28T05:53:49.821106Z","shell.execute_reply":"2024-09-28T06:15:43.986004Z"},"trusted":true},"execution_count":1,"outputs":[{"name":"stdout","text":"Cloning into 'llama.cpp'...\nremote: Enumerating objects: 34888, done.\u001b[K\nremote: Counting objects: 100% (6871/6871), done.\u001b[K\nremote: Compressing objects: 100% (431/431), done.\u001b[K\nremote: Total 34888 (delta 6647), reused 6531 (delta 6436), pack-reused 28017 (from 1)\u001b[K\nReceiving objects: 100% (34888/34888), 58.34 MiB | 25.22 MiB/s, done.\nResolving deltas: 100% (25302/25302), done.\n/kaggle/working/llama.cpp\n","output_type":"stream"}]},{"cell_type":"code","source":"!chmod +x -R llama.cpp\n%cd llama.cpp\n\n","metadata":{"execution":{"iopub.status.busy":"2024-09-28T06:15:43.991592Z","iopub.execute_input":"2024-09-28T06:15:43.992153Z","iopub.status.idle":"2024-09-28T06:17:23.223817Z","shell.execute_reply.started":"2024-09-28T06:15:43.992088Z","shell.execute_reply":"2024-09-28T06:17:23.222710Z"},"trusted":true},"execution_count":2,"outputs":[{"name":"stdout","text":"NOTICE: The 'server' binary is deprecated. Please use 'llama-server' instead.\n\u001b[0mSuccessfully installed MarkupSafe-2.1.5 certifi-2024.8.30 charset-normalizer-3.3.2 filelock-3.16.1 fsspec-2024.9.0 gguf-0.10.0 huggingface-hub-0.25.1 idna-3.10 jinja2-3.1.4 mpmath-1.3.0 networkx-3.3 numpy-1.26.4 packaging-24.1 protobuf-4.25.3 pyyaml-6.0.2 regex-2024.9.11 requests-2.32.3 safetensors-0.4.5 sentencepiece-0.2.0 sympy-1.13.3 tokenizers-0.20.0 torch-2.2.2+cpu tqdm-4.66.5 transformers-4.45.1 typing-extensions-4.12.2 urllib3-2.2.1\n","output_type":"stream"}]},{"cell_type":"code","source":"!cp -r /usr/local/cuda-12.3/targets /usr/local/nvidia/ ","metadata":{"execution":{"iopub.status.busy":"2024-09-28T06:17:23.225233Z","iopub.execute_input":"2024-09-28T06:17:23.225537Z","iopub.status.idle":"2024-09-28T06:17:23.230915Z","shell.execute_reply.started":"2024-09-28T06:17:23.225504Z","shell.execute_reply":"2024-09-28T06:17:23.229928Z"},"trusted":true},"execution_count":3,"outputs":[]},{"cell_type":"markdown","source":"# Run perplexity\nPerplexity is the main evaluation metric for large language models (LLMs). It measures how well the model predicts a given sequence of tokens. Formally, the perplexity of LLMs is the exponentiated average negative log-likelihood, which means perplexity smaller is better\n\nPerplexity benchmark shows how accurate the model is","metadata":{}},{"cell_type":"code","source":"!/kaggle/working/llama.cpp/llama-perplexity -m /kaggle/input/all-minilm-l6-v2/gguf/fp16/1/all-MiniLM-L6-v2.gguf -f /kaggle/working/llama.cpp/wikitext-2-raw/wiki.test.raw --chunks 256 > ppl_L6_fp16.log 2>&1\n!/kaggle/working/llama.cpp/llama-perplexity -m /kaggle/input/all-minilm-l6-v2/gguf/q4_k_m/1/all-MiniLM-L6-v2-Q4_K_M-v2.gguf -f /kaggle/working/llama.cpp/wikitext-2-raw/wiki.test.raw --chunks 256 > ppl_Q4_v2.log 2>&1\n!/kaggle/working/llama.cpp/llama-perplexity -m /kaggle/input/all-minilm-l6-v2/gguf/q4_k_m/1/all-MiniLM-L6-v2-Q4_K_M.gguf -f /kaggle/working/llama.cpp/wikitext-2-raw/wiki.test.raw --chunks 256 > ppl_Q4.log 2>&1\n","metadata":{"execution":{"iopub.status.busy":"2024-09-28T06:44:13.360373Z","iopub.execute_input":"2024-09-28T06:44:13.360782Z"},"trusted":true},"execution_count":null,"outputs":[{"name":"stdout","text":"^C\n^C\n","output_type":"stream"}]},{"cell_type":"code","source":"!tail -6 ppl_L6_fp16.log\n!tail -6 ppl_Q4_v2.log\n!tail -6 ppl_Q4.log","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Benchmarking the Inference Throughput and Memory Consumption\nLlama-bench shows performance and resource consumption. There are three types of test\n\n- Prompt processing (pp): processing a prompt in batches (-p)\n- Text generation (tg): generating a sequence of tokens (-n)\n- Prompt processing + text generation (pg): processing a prompt followed by generating a sequence of tokens (-pg)\nIn the following test, we run text generation and prompt processing","metadata":{}},{"cell_type":"code","source":"!/kaggle/working/llama.cpp/llama-bench -m /kaggle/input/all-minilm-l6-v2/gguf/fp16/1/all-MiniLM-L6-v2.gguf -m /kaggle/input/all-minilm-l6-v2/gguf/q4_k_m/1/all-MiniLM-L6-v2-Q4_K_M-v2.gguf -m /kaggle/input/all-minilm-l6-v2/gguf/q4_k_m/1/all-MiniLM-L6-v2-Q4_K_M.gguf -mg 1 -ngl 10,20,30,35 -n 128,256,512 ","metadata":{"execution":{"iopub.status.busy":"2024-09-28T06:28:59.330210Z","iopub.execute_input":"2024-09-28T06:28:59.331034Z","iopub.status.idle":"2024-09-28T06:29:00.670964Z","shell.execute_reply.started":"2024-09-28T06:28:59.330998Z","shell.execute_reply":"2024-09-28T06:29:00.669691Z"},"trusted":true},"execution_count":7,"outputs":[{"name":"stdout","text":"ggml_cuda_init: GGML_CUDA_FORCE_MMQ:    no\nggml_cuda_init: GGML_CUDA_FORCE_CUBLAS: no\nggml_cuda_init: found 1 CUDA devices:\n  Device 0: Tesla P100-PCIE-16GB, compute capability 6.0, VMM: yes\n| model                          |       size |     params | backend    | ngl |   main_gpu |          test |                  t/s |\n| ------------------------------ | ---------: | ---------: | ---------- | --: | ---------: | ------------: | -------------------: |\n| bert 22M F16                   |  43.48 MiB |    22.57 M | CUDA       |  10 |          1 |         pp512 | 30833069.70 ± 6161731.76 |\nsrc/llama.cpp:16900: GGML_ASSERT(strcmp(res->name, \"result_output\") == 0 && \"missing result_output tensor\") failed\n","output_type":"stream"}]}]}