Python: Verify local models in Ollama and LM Studio are compatible with the OpenAI connector #6973
Conversation
markwallace-microsoft
added
the
python
github-actions
bot
changed the title
Python 3.10 Test Coverage Report •
Python 3.10 Unit Test Overview
|
There was a problem hiding this comment.
When testing this sample locally with semantic-kernel==1.1.2, I needed to add an api_key (fake) to line 39 OpenAIChatCompletion as well. Otherwise I get a parameter validation error. Maybe I'm wrong, but please verify.
There was a problem hiding this comment.
That shouldn't happen as the api_key is not a model field. Could you provide more information, such as the validation error message?
There was a problem hiding this comment.
Running your ollama_chat_completion.py with semantic-kernel 1.1.2, openai 1.35.7 on macOS 14.5 produces the following Traceback (most recent call last): Note, I had to change phi3 to phi3:mini for my computer.
File "/opt/homebrew/Caskroom/miniconda/base/envs/sk/lib/python3.12/site-packages/semantic_kernel/connectors/ai/open_ai/services/open_ai_chat_completion.py", line 51, in init
openai_settings = OpenAISettings.create(
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/sk/lib/python3.12/site-packages/semantic_kernel/kernel_pydantic.py", line 56, in create
return cls(**data)
^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/sk/lib/python3.12/site-packages/pydantic_settings/main.py", line 140, in init
super().init(
File "/opt/homebrew/Caskroom/miniconda/base/envs/sk/lib/python3.12/site-packages/pydantic/main.py", line 176, in init
self.pydantic_validator.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for OpenAISettings
api_key
Field required [type=missing, input_value={'chat_model_id': 'phi3:mini'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.7/v/missing
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/andi/Projects/sk/ollama_chat_completion.py", line 39, in
kernel.add_service(OpenAIChatCompletion(service_id=service_id, ai_model_id="phi3:mini", async_client=openAIClient))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/sk/lib/python3.12/site-packages/semantic_kernel/connectors/ai/open_ai/services/open_ai_chat_completion.py", line 59, in init
raise ServiceInitializationError("Failed to create OpenAI settings.", ex) from ex
semantic_kernel.exceptions.service_exceptions.ServiceInitializationError: ('Failed to create OpenAI settings.', 1 validation error for OpenAISettings
api_key
Field required [type=missing, input_value={'chat_model_id': 'phi3:mini'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.7/v/missing)
If I change Line 39 by adding the fake API key as 2nd argument (to kernel.add_service(OpenAIChatCompletion(service_id=service_id, api_key="fake-key", ...) the error goes away and the code works.
Hope this helps, Andi
There was a problem hiding this comment.
Hi @TaoChenOSU, yes, this is true as @AndreasKunar pointed out. In the OpenAISettings the model has the api_key as a required attribute. See here.
For Ollama local models that don't require an API key, and it is not provided, this will fail. Which is why Andi needed to provide a fake key to get this model validation to pass.
There was a problem hiding this comment.
I see. Then I guess the api key shouldn't be a required parameter but if one is not provided and no async_client is provided, it will be caught here.
There was a problem hiding this comment.
I was able to run the sample without error because I had the env file.
There was a problem hiding this comment.
Should we make the change to the OpenAI settings?
There was a problem hiding this comment.
Running your ollama_chat_completion.py with semantic-kernel 1.1.2, openai 1.35.7 on macOS 14.5 produces the following Traceback (most recent call last): Note, I had to change phi3 to phi3:mini for my computer.
File "/opt/homebrew/Caskroom/miniconda/base/envs/sk/lib/python3.12/site-packages/semantic_kernel/connectors/ai/open_ai/services/open_ai_chat_completion.py", line 51, in init
openai_settings = OpenAISettings.create(
^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/sk/lib/python3.12/site-packages/semantic_kernel/kernel_pydantic.py", line 56, in create
return cls(**data)
^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/sk/lib/python3.12/site-packages/pydantic_settings/main.py", line 140, in init
super().init(
File "/opt/homebrew/Caskroom/miniconda/base/envs/sk/lib/python3.12/site-packages/pydantic/main.py", line 176, in init
self.pydantic_validator.validate_python(data, self_instance=self)
pydantic_core._pydantic_core.ValidationError: 1 validation error for OpenAISettings
api_key
Field required [type=missing, input_value={'chat_model_id': 'phi3:mini'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.7/v/missing
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File "/Users/andi/Projects/sk/ollama_chat_completion.py", line 39, in
kernel.add_service(OpenAIChatCompletion(service_id=service_id, ai_model_id="phi3:mini", async_client=openAIClient))
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "/opt/homebrew/Caskroom/miniconda/base/envs/sk/lib/python3.12/site-packages/semantic_kernel/connectors/ai/open_ai/services/open_ai_chat_completion.py", line 59, in init
raise ServiceInitializationError("Failed to create OpenAI settings.", ex) from ex
semantic_kernel.exceptions.service_exceptions.ServiceInitializationError: ('Failed to create OpenAI settings.', 1 validation error for OpenAISettings
api_key
Field required [type=missing, input_value={'chat_model_id': 'phi3:mini'}, input_type=dict]
For further information visit https://errors.pydantic.dev/2.7/v/missing)
If I change Line 39 by adding the fake API key as 2nd argument (to kernel.add_service(OpenAIChatCompletion(service_id=service_id, api_key="fake-key", ...) the error goes away and the code works.
Hope this helps, Andi
Motivation and Context
Related to #6498
The use of local models presents a twofold benefit for developers: increased flexibility and reduced costs. Ollama and LM Studio are two well-known platforms that facilitate the hosting of models locally, both of which offer compatibility with OpenAI endpoints. As such, it is imperative that our OpenAI connector functions correctly when users are operating models on these platforms.
Description
/concepts/local_modelsto show how to using local models with the OpenAI connector.test_sample_utils.pywhere if a test case is retried and input was never reset.Contribution Checklist