From e418e316d0aed4c36aad64103bd09c802c5f2ec2 Mon Sep 17 00:00:00 2001 From: Natan Yellin Date: Fri, 31 May 2024 17:44:14 +0300 Subject: [PATCH 1/3] Add warning regarding function calling and self-hosted models --- README.md | 7 ++++++- 1 file changed, 6 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index 18ab1f1787..c34a5b020d 100644 --- a/README.md +++ b/README.md @@ -166,7 +166,12 @@ holmes ask "what pods are unhealthy and why?" --llm=azure --api-key= Using a self-hosted LLM like llama 3 -You will need an LLM with support for tool-calling. To use it, set the OPENAI_BASE_URL environment variable and run `holmes` with a relevant model name set using `--model`. +You will need an LLM with support for function-calling (tool-calling). To use it, set the OPENAI_BASE_URL environment variable and run `holmes` with a relevant model name set using `--model`. + +**Important: Please verify that your model and inference server support function calling! HolmesGPT is currently unable to check if the LLM it was given supports function-calling or not. Some models that lack function-calling capabilities will hallucinate answers instead of reporting that they are unable to call functions. This behaviour depends on the model.** + +In particular, note that [vLLM does yet support function calling](https://github.com/vllm-project/vllm/issues/1869), whereas [llama-cpp does support it[(https://github.com/abetlen/llama-cpp-python?tab=readme-ov-file#function-calling). + ## More Examples From 92ba710b6722478b8d3c6d81cfa8292d50ac61e4 Mon Sep 17 00:00:00 2001 From: Natan Yellin Date: Fri, 31 May 2024 17:45:07 +0300 Subject: [PATCH 2/3] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index c34a5b020d..f202ffd56f 100644 --- a/README.md +++ b/README.md @@ -164,7 +164,7 @@ holmes ask "what pods are unhealthy and why?" --llm=azure --api-key=
-Using a self-hosted LLM like llama 3 +Using a self-hosted LLM You will need an LLM with support for function-calling (tool-calling). To use it, set the OPENAI_BASE_URL environment variable and run `holmes` with a relevant model name set using `--model`. From e246b2f81d6fd56497080abb44de0dffa18d3403 Mon Sep 17 00:00:00 2001 From: Natan Yellin Date: Fri, 31 May 2024 17:45:59 +0300 Subject: [PATCH 3/3] Update README.md --- README.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/README.md b/README.md index f202ffd56f..e7d1cb18ec 100644 --- a/README.md +++ b/README.md @@ -170,7 +170,7 @@ You will need an LLM with support for function-calling (tool-calling). To use it **Important: Please verify that your model and inference server support function calling! HolmesGPT is currently unable to check if the LLM it was given supports function-calling or not. Some models that lack function-calling capabilities will hallucinate answers instead of reporting that they are unable to call functions. This behaviour depends on the model.** -In particular, note that [vLLM does yet support function calling](https://github.com/vllm-project/vllm/issues/1869), whereas [llama-cpp does support it[(https://github.com/abetlen/llama-cpp-python?tab=readme-ov-file#function-calling). +In particular, note that [vLLM does yet support function calling](https://github.com/vllm-project/vllm/issues/1869), whereas [llama-cpp does support it](https://github.com/abetlen/llama-cpp-python?tab=readme-ov-file#function-calling).