Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Python: Support Auto tool calling for OpenAI models #5267

Merged
merged 29 commits into from
Mar 5, 2024
Merged

Python: Support Auto tool calling for OpenAI models #5267

merged 29 commits into from
Mar 5, 2024

Conversation

moonbox3
Copy link
Contributor

@moonbox3 moonbox3 commented Mar 1, 2024

Motivation and Context

Auto tool calling hasn't been implemented in Python for OpenAI models. This is a much needed feature, which helps open the possibilities and extends being able to use the OpenAI models for just pure back and forth chats.

Description

This PR:

  • introduces auto tool calls for OpenAI models. If the prompt function is of an OpenAI chat completion type, then we pass the kernel as an argument to the OpenAI Chat Completion Base class. The kernel is used to auto-invoke tools/functions seamlessly as part of the chat completion request. Closes Python: Enable auto function/tool calls #4553
  • splits the current chat completion/chat completion stream methods into smaller, more readable/functional methods in the OpenAI Chat Completion Base class.
  • Adds an integration test to cover auto invoke tool calls
  • Updates a kernel example to be able to work for a streaming/non-streaming tool call. Other Kernel Examples that don't use auto invoke tools still work as they did before.
  • Removes old code in the utils file that was related to the manual form of invoking function calls.

Contribution Checklist

@moonbox3 moonbox3 requested a review from a team as a code owner March 1, 2024 19:03
@markwallace-microsoft markwallace-microsoft added the python Pull requests for the Python Semantic Kernel label Mar 1, 2024
@markwallace-microsoft
Copy link
Member

markwallace-microsoft commented Mar 1, 2024 •
edited
Loading

Tests Skipped Failures Errors Time
978 11 💤 0 ❌ 0 🔥 19.403s ⏱️

eavanvalkenburg
eavanvalkenburg requested changes Mar 1, 2024
View reviewed changes
Copy link
Member

@eavanvalkenburg eavanvalkenburg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking great overall! Some notes added!

python/semantic_kernel/connectors/ai/google_palm/services/gp_chat_completion.py Outdated Show resolved Hide resolved
python/semantic_kernel/connectors/ai/ollama/services/ollama_chat_completion.py Outdated Show resolved Hide resolved
python/semantic_kernel/connectors/ai/open_ai/services/open_ai_chat_completion_base.py Show resolved Hide resolved
python/semantic_kernel/connectors/ai/open_ai/services/open_ai_chat_completion_base.py Show resolved Hide resolved
python/semantic_kernel/connectors/ai/open_ai/services/open_ai_chat_completion_base.py Outdated Show resolved Hide resolved
python/semantic_kernel/connectors/ai/open_ai/services/open_ai_chat_completion_base.py Outdated Show resolved Hide resolved
python/semantic_kernel/contents/chat_history.py Show resolved Hide resolved
@markwallace-microsoft
Copy link
Member

Python Test Coverage

Python Test Coverage Report •
FileStmtsMissCoverMissing
semantic_kernel
   kernel.py3107775%127, 143–147, 181–193, 197–201, 203–207, 209–213, 220–222, 241, 247, 249–253, 255–259, 261–266, 268, 303–307, 324–329, 336, 340–344, 357, 382–390, 434–435, 438–439, 479, 485, 501, 519, 535, 554, 566, 599–603, 638, 723, 725–731, 745–746
semantic_kernel/connectors/ai
   chat_completion_client_base.py9189%18
semantic_kernel/connectors/ai/ollama/services
   ollama_chat_completion.py62297%110, 176
semantic_kernel/connectors/ai/open_ai
   utils.py97970%3–303
semantic_kernel/connectors/ai/open_ai/contents
   open_ai_streaming_chat_message_content.py281739%48–67
   tool_calls.py11373%17–19
semantic_kernel/connectors/ai/open_ai/services
   open_ai_chat_completion_base.py985643%88–106, 112–114, 131–133, 147–150, 163–183, 196, 204, 210–216, 227–233
semantic_kernel/contents
   chat_history.py116992%108, 244–245, 276–278, 290–292
semantic_kernel/functions
   kernel_function_from_prompt.py92199%194
semantic_kernel/prompt_template
   prompt_template_config.py51590%47, 69, 74–75, 83
TOTAL4781105178% 

Python Unit Test Overview

Tests Skipped Failures Errors Time
978 11 💤 0 ❌ 0 🔥 19.805s ⏱️

@markwallace-microsoft
Copy link
Member

Python Test Coverage

Python Test Coverage Report •
FileStmtsMissCoverMissing
semantic_kernel
   kernel.py3107775%127, 143–147, 181–193, 197–201, 203–207, 209–213, 220–222, 241, 247, 249–253, 255–259, 261–266, 268, 303–307, 324–329, 336, 340–344, 357, 382–390, 434–435, 438–439, 479, 485, 501, 519, 535, 554, 566, 599–603, 638, 723, 725–731, 745–746
semantic_kernel/connectors/ai
   chat_completion_client_base.py9189%18
semantic_kernel/connectors/ai/ollama/services
   ollama_chat_completion.py62297%110, 176
semantic_kernel/connectors/ai/open_ai
   utils.py97970%3–303
semantic_kernel/connectors/ai/open_ai/contents
   open_ai_streaming_chat_message_content.py281739%48–67
   tool_calls.py11373%17–19
semantic_kernel/connectors/ai/open_ai/services
   open_ai_chat_completion_base.py985643%88–106, 112–114, 131–133, 147–150, 163–183, 196, 204, 210–216, 227–233
semantic_kernel/contents
   chat_history.py116992%108, 244–245, 276–278, 290–292
semantic_kernel/functions
   kernel_function_from_prompt.py92199%194
semantic_kernel/prompt_template
   prompt_template_config.py51590%47, 69, 74–75, 83
TOTAL4781105178% 

Python Unit Test Overview

Tests Skipped Failures Errors Time
978 11 💤 0 ❌ 0 🔥 18.465s ⏱️

@markwallace-microsoft
Copy link
Member

Python Test Coverage

Python Test Coverage Report •
FileStmtsMissCoverMissing
semantic_kernel
   kernel.py3107775%127, 143–147, 181–193, 197–201, 203–207, 209–213, 220–222, 241, 247, 249–253, 255–259, 261–266, 268, 303–307, 324–329, 336, 340–344, 357, 382–390, 434–435, 438–439, 479, 485, 501, 519, 535, 554, 566, 599–603, 638, 723, 725–731, 745–746
semantic_kernel/connectors/ai
   chat_completion_client_base.py9189%18
semantic_kernel/connectors/ai/ollama/services
   ollama_chat_completion.py62297%110, 176
semantic_kernel/connectors/ai/open_ai
   utils.py97970%3–303
semantic_kernel/connectors/ai/open_ai/contents
   open_ai_streaming_chat_message_content.py281739%48–67
   tool_calls.py11373%17–19
semantic_kernel/connectors/ai/open_ai/services
   open_ai_chat_completion_base.py985643%88–106, 112–114, 131–133, 147–150, 163–183, 196, 204, 210–216, 227–233
semantic_kernel/contents
   chat_history.py116992%108, 244–245, 276–278, 290–292
semantic_kernel/functions
   kernel_function_from_prompt.py92199%194
semantic_kernel/prompt_template
   prompt_template_config.py51590%47, 69, 74–75, 83
TOTAL4781105178% 

Python Unit Test Overview

Tests Skipped Failures Errors Time
978 11 💤 0 ❌ 0 🔥 17.903s ⏱️

@markwallace-microsoft
Copy link
Member

Python Test Coverage

Python Test Coverage Report •
FileStmtsMissCoverMissing
semantic_kernel
   kernel.py3107775%113, 129–133, 167–179, 183–187, 189–193, 195–199, 206–208, 227, 233, 235–239, 241–245, 247–252, 254, 289–293, 310–315, 322, 326–330, 343, 368–376, 420–421, 424–425, 465, 471, 487, 505, 521, 540, 552, 585–589, 627, 712, 714–720, 734–735
semantic_kernel/connectors/ai
   chat_completion_client_base.py9189%18
semantic_kernel/connectors/ai/ollama/services
   ollama_chat_completion.py62297%110, 176
semantic_kernel/connectors/ai/open_ai
   utils.py97970%3–303
semantic_kernel/connectors/ai/open_ai/contents
   open_ai_streaming_chat_message_content.py281739%48–67
   tool_calls.py11373%17–19
semantic_kernel/connectors/ai/open_ai/services
   open_ai_chat_completion_base.py985643%88–106, 112–114, 131–133, 147–150, 163–183, 196, 204, 210–216, 227–233
semantic_kernel/contents
   chat_history.py116992%108, 244–245, 276–278, 290–292
semantic_kernel/functions
   kernel_function_from_prompt.py92199%194
semantic_kernel/prompt_template
   prompt_template_config.py51590%47, 69, 74–75, 83
TOTAL4781105178% 

Python Unit Test Overview

Tests Skipped Failures Errors Time
978 11 💤 0 ❌ 0 🔥 25.579s ⏱️

@markwallace-microsoft
Copy link
Member

Python Test Coverage

Python Test Coverage Report •
FileStmtsMissCoverMissing
semantic_kernel
   kernel.py3107775%113, 129–133, 167–179, 183–187, 189–193, 195–199, 206–208, 227, 233, 235–239, 241–245, 247–252, 254, 289–293, 310–315, 322, 326–330, 343, 368–376, 420–421, 424–425, 465, 471, 487, 505, 521, 540, 552, 585–589, 627, 712, 714–720, 734–735
semantic_kernel/connectors/ai
   chat_completion_client_base.py9189%18
semantic_kernel/connectors/ai/ollama/services
   ollama_chat_completion.py62297%110, 176
semantic_kernel/connectors/ai/open_ai
   utils.py97970%3–303
semantic_kernel/connectors/ai/open_ai/contents
   open_ai_streaming_chat_message_content.py281739%48–67
   tool_calls.py11373%17–19
semantic_kernel/connectors/ai/open_ai/services
   open_ai_chat_completion_base.py985643%88–106, 112–114, 131–133, 147–150, 163–183, 196, 204, 210–216, 227–233
semantic_kernel/contents
   chat_history.py116992%108, 244–245, 276–278, 290–292
semantic_kernel/functions
   kernel_function_from_prompt.py92199%194
semantic_kernel/prompt_template
   prompt_template_config.py51590%47, 69, 74–75, 83
TOTAL4781105178% 

Python Unit Test Overview

Tests Skipped Failures Errors Time
978 11 💤 0 ❌ 0 🔥 16.815s ⏱️

@markwallace-microsoft
Copy link
Member

Python Test Coverage

Python Test Coverage Report •
FileStmtsMissCoverMissing
semantic_kernel
   kernel.py3107775%113, 129–133, 167–179, 183–187, 189–193, 195–199, 206–208, 227, 233, 235–239, 241–245, 247–252, 254, 289–293, 310–315, 322, 326–330, 343, 368–376, 420–421, 424–425, 465, 471, 487, 505, 521, 540, 552, 585–589, 627, 712, 714–720, 734–735
semantic_kernel/connectors/ai
   chat_completion_client_base.py9189%18
semantic_kernel/connectors/ai/ollama/services
   ollama_chat_completion.py62297%110, 176
semantic_kernel/connectors/ai/open_ai
   utils.py97970%3–303
semantic_kernel/connectors/ai/open_ai/contents
   open_ai_streaming_chat_message_content.py281739%48–67
   tool_calls.py11373%17–19
semantic_kernel/connectors/ai/open_ai/services
   open_ai_chat_completion_base.py985643%88–106, 112–114, 131–133, 147–150, 163–183, 196, 204, 210–216, 227–233
semantic_kernel/contents
   chat_history.py116992%108, 244–245, 276–278, 290–292
semantic_kernel/functions
   kernel_function_from_prompt.py92199%194
semantic_kernel/prompt_template
   prompt_template_config.py51590%47, 69, 74–75, 83
TOTAL4781105178% 

Python Unit Test Overview

Tests Skipped Failures Errors Time
978 11 💤 0 ❌ 0 🔥 20.019s ⏱️

@markwallace-microsoft
Copy link
Member

Python Test Coverage

Python Test Coverage Report •
FileStmtsMissCoverMissing
semantic_kernel
   kernel.py3107775%113, 129–133, 167–179, 183–187, 189–193, 195–199, 206–208, 227, 233, 235–239, 241–245, 247–252, 254, 289–293, 310–315, 322, 326–330, 343, 368–376, 420–421, 424–425, 465, 471, 487, 505, 521, 540, 552, 585–589, 627, 712, 714–720, 734–735
semantic_kernel/connectors/ai
   chat_completion_client_base.py9189%18
semantic_kernel/connectors/ai/ollama/services
   ollama_chat_completion.py62297%110, 176
semantic_kernel/connectors/ai/open_ai
   utils.py97970%3–303
semantic_kernel/connectors/ai/open_ai/contents
   open_ai_streaming_chat_message_content.py281739%48–67
   tool_calls.py11373%17–19
semantic_kernel/connectors/ai/open_ai/services
   open_ai_chat_completion_base.py985643%88–106, 112–114, 131–133, 147–150, 163–183, 196, 204, 210–216, 227–233
semantic_kernel/contents
   chat_history.py116992%108, 244–245, 276–278, 290–292
semantic_kernel/functions
   kernel_function_from_prompt.py92199%194
semantic_kernel/prompt_template
   prompt_template_config.py51590%47, 69, 74–75, 83
TOTAL4781105178% 

Python Unit Test Overview

Tests Skipped Failures Errors Time
978 11 💤 0 ❌ 0 🔥 23.103s ⏱️

@moonbox3
Copy link
Contributor Author

moonbox3 commented Mar 5, 2024

Going to add some more unit tests.

eavanvalkenburg
eavanvalkenburg approved these changes Mar 5, 2024
View reviewed changes
Copy link
Member

@eavanvalkenburg eavanvalkenburg left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good, made one small note, but this works as well!

python/semantic_kernel/connectors/ai/chat_completion_client_base.py Show resolved Hide resolved
juliomenendez
juliomenendez approved these changes Mar 5, 2024
View reviewed changes
Copy link
Contributor

@juliomenendez juliomenendez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looking good, just a couple comments but not blocking the PR because of them.

python/semantic_kernel/connectors/ai/open_ai/services/open_ai_chat_completion_base.py Outdated Show resolved Hide resolved
python/semantic_kernel/connectors/ai/open_ai/services/open_ai_chat_completion_base.py Outdated Show resolved Hide resolved
juliomenendez
juliomenendez approved these changes Mar 5, 2024
View reviewed changes
Copy link
Contributor

@juliomenendez juliomenendez left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@moonbox3 moonbox3 enabled auto-merge March 5, 2024 20:32
@moonbox3 moonbox3 added this pull request to the merge queue Mar 5, 2024
github-merge-queue bot pushed a commit that referenced this pull request Mar 5, 2024
@moonbox3 @eavanvalkenburg
Python: Support Auto tool calling for OpenAI models (#5267)
c546de6 
### Motivation and Context

Auto tool calling hasn't been implemented in Python for OpenAI models.
This is a much needed feature, which helps open the possibilities and
extends being able to use the OpenAI models for just pure back and forth
chats.

<!-- Thank you for your contribution to the semantic-kernel repo!
Please help reviewers and future users, providing the following
information:
  1. Why is this change required?
  2. What problem does it solve?
  3. What scenario does it contribute to?
  4. If it fixes an open issue, please link to the issue here.
-->

### Description

This PR:
- introduces auto tool calls for OpenAI models. If the prompt function
is of an OpenAI chat completion type, then we pass the kernel as an
argument to the OpenAI Chat Completion Base class. The kernel is used to
auto-invoke tools/functions seamlessly as part of the chat completion
request. Closes #4553
- splits the current chat completion/chat completion stream methods into
smaller, more readable/functional methods in the OpenAI Chat Completion
Base class.
- Adds an integration test to cover auto invoke tool calls
- Updates a kernel example to be able to work for a
streaming/non-streaming tool call. Other Kernel Examples that don't use
auto invoke tools still work as they did before.
- Removes old code in the utils file that was related to the manual form
of invoking function calls.

<!-- Describe your changes, the overall approach, the underlying design.
These notes will help understanding how your code works. Thanks! -->

### Contribution Checklist

<!-- Before submitting this PR, please make sure: -->

- [X] The code builds clean without any errors or warnings
- [X] The PR follows the [SK Contribution
Guidelines](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md)
and the [pre-submission formatting
script](https://github.com/microsoft/semantic-kernel/blob/main/CONTRIBUTING.md#development-scripts)
raises no violations
- [X] All unit tests pass, and I have added new tests where possible
- [X] I didn't break anyone 😄

---------

Co-authored-by: Eduard van Valkenburg <[email protected]>
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 5, 2024
@moonbox3 moonbox3 added this pull request to the merge queue Mar 5, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 5, 2024
@moonbox3 moonbox3 added this pull request to the merge queue Mar 5, 2024
@github-merge-queue github-merge-queue bot removed this pull request from the merge queue due to failed status checks Mar 5, 2024
@moonbox3 moonbox3 enabled auto-merge March 5, 2024 21:12
@moonbox3 moonbox3 added this pull request to the merge queue Mar 5, 2024
Merged via the queue into microsoft:main with commit b93bab9 Mar 5, 2024
27 checks passed
@moonbox3 moonbox3 deleted the auto_function_calling branch March 5, 2024 21:18
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@juliomenendez juliomenendez juliomenendez approved these changes

@eavanvalkenburg eavanvalkenburg eavanvalkenburg approved these changes

Assignees
No one assigned
Labels
documentation python Pull requests for the Python Semantic Kernel
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Python: Enable auto function/tool calls
4 participants
@moonbox3 @markwallace-microsoft @juliomenendez @eavanvalkenburg