Python: Feat/Add DEVELOPER role for openai o1

[ymuichiro]

Motivation and Context

Currently, the OpenAI O1 model introduces a new role "developer". However, passing the conventional "system" role results in an error. To address this issue, the following changes are proposed.

Description

1. Add a Method to the ChatHistory Class

   • Implement a method to handle role conversion or substitution logic for compatibility with the O1 model.

2. Expand the AuthorRole Enum

   • Add "developer" as a new value to the AuthorRole enum.

3. Improve Developer Experience (UX)

   • If the "system" role is mistakenly passed, the logic should internally convert it to "developer" for better UX.
   • However, since all models other than O1 still use "system", simply adding support for the "developer" role seems to be the most optimal solution at this point.

• Closes: Python: New Feature: Add Support for "developer" Role for OpenAI o1 model #10032

Background

• The O1 model no longer supports the "system" role, causing errors when it is passed.
• The introduction of the "developer" role must be integrated in a way that preserves compatibility with other models.

Usage

import asyncio

from semantic_kernel.connectors.ai.function_choice_behavior import FunctionChoiceBehavior
from semantic_kernel.connectors.ai.open_ai.prompt_execution_settings.azure_chat_prompt_execution_settings import (
    AzureChatPromptExecutionSettings,
)
from semantic_kernel.connectors.ai.open_ai.services.azure_chat_completion import AzureChatCompletion
from semantic_kernel.contents.chat_history import ChatHistory
from semantic_kernel.kernel import Kernel

async def main():
    kernel = Kernel()
    service_id = "azure-openai"
    kernel.add_service(
        service=AzureChatCompletion(
            service_id=service_id,
            api_key=*********,
            endpoint=*********,
            deployment_name="o1",
            api_version="2024-12-01-preview",
        )
    )

    settings = kernel.get_prompt_execution_settings_from_service_id(service_id=service_id)

    if isinstance(settings, AzureChatPromptExecutionSettings):
        settings.function_choice_behavior = FunctionChoiceBehavior.Auto(auto_invoke=True)

    service = kernel.get_service(service_id=service_id)

    if not isinstance(service, AzureChatCompletion):
        raise Exception("Invalid Value")

    history = ChatHistory()
    history.add_developer_message("You are a helpful assistant.")
    history.add_user_message("hello")

    result = await service.get_chat_message_contents(chat_history=history, settings=settings, kernel=kernel)

    if not result:
        raise Exception("result is None")

    print(result[0].content)

if __name__ == "__main__":
    asyncio.run(main())

Contribution Checklist

• The code builds clean without any errors or warnings
• The PR follows the SK Contribution Guidelines and the pre-submission formatting script raises no violations
• All unit tests pass, and I have added new tests where possible
• I didn't break anyone 😄

[moonbox3]

Thanks for having a look at this, @ymuichiro. I think it’d be good to make sure other o1 api features are in as well. I saw they have a reasoning_effort param, too. Are there other features we’d want to get in to make sure the o1 experience works well?

[ymuichiro]

@moonbox3

Thank you for checking this so quickly. Regarding o1, here are the changes I’m aware of:

※ It might be worth considering providing OpenAIChatReasoningBase as a separate service from OpenAIChatCompletionBase in the future. However, since it seems that the functionality is being gradually integrated, I believe the appropriate place to incorporate these changes for now would be in OpenAIChatPromptExecutionSettings.

o1: 2024-12-17
API: 2024-09-01-preview

1. In o1, the system role has been deprecated, and a new developer role has been introduced.
2. In o1, max_tokens has been deprecated and replaced with max_completion_tokens.
   • max_completion_tokens: int
   • The value represents the sum of reasoning tokens and completion tokens, with a recommendation to specify 25,000 or more.
3. A new property, reasoning_effort, has been added in o1.
   • reasoning_effort: enum, low, medium, high
   • Higher values indicate that the model will spend more time processing the request.
4. A new usage metric, reasoning_tokens, has been introduced in o1.
   • This represents the number of tokens used during reasoning.

{
  "completion_tokens": 1843,
  "prompt_tokens": 20,
  "total_tokens": 1863,
  "completion_tokens_details": {
    "audio_tokens": null,
    "reasoning_tokens": 448
  },
  "prompt_tokens_details": {
    "audio_tokens": null,
    "cached_tokens": 0
  }
}

Currently Unsupported

• Parallel tool invocation
• temperature, top_p, presence_penalty, frequency_penalty, logprobs, top_logprobs, logit_bias

It would be nice if these were applied to settings, usage, and author_role.

[eavanvalkenburg]

I agree @moonbox3 bit more work is needed to make sure we fully support o1, these extra settings can just be added with None defaults, but we also need to consider the rendered prompt to ChatHistory parser, that uses a system message as default, that can be left as is, but might throw when used with o1 but might also be usefull to have a look at how we can support this as well!

[ymuichiro]

@moonbox3 @eavanvalkenburg

Thank you for your comment. After reviewing the current Semantic Kernel code, I found several instances where AuthorRole.SYSTEM is used as a default value. Since these involve shared classes that are used not only with OpenAI models but also with other models, I believe it is not appropriate to make changes that could impact them. Additionally, to handle these cases comprehensively, it is necessary to determine whether the model in question is a traditional ChatCompletion model or a Reasoning model.

With that in mind, I am planning to proceed with the following tasks. Does this approach make sense to you?

1. Support for the new role AuthorRole.DEVELOPER

   Add a new inference class to automatically replace AuthorRole.SYSTEM with AuthorRole.DEVELOPER where specified.

   • Additions to model types:

     • Add REASONING to OpenAIModelTypes in semantic_kernel/connectors/ai/open_ai/services/open_ai_model_types.py.

   • Additions to inference classes:

     • Add the following three classes extending OpenAIChatCompletionBase in semantic_kernel/connectors/ai/open_ai/services/open_ai_chat_completion_base.py:
       • OpenAIChatReasoning
       • AzureChatReasoning
     • Add the following class extending AzureAIInferenceBase in semantic_kernel/connectors/ai/azure_ai_inference/services/azure_ai_inference_base.py:
       • AzureAIInferenceChatReasoning

※ Most methods can be reused, so it might be sufficient to simply override some methods of the existing OpenAIChatCompletion (...etc) class.

2. Addition of new configuration parameters:
   1. max_completion_tokens
   2. reasoning_effort

Regarding item 2, even if certain properties cannot be used, I believe it is the responsibility of the user to ensure only valid properties are passed. Therefore, I plan to set the default values for these properties to None, so they will only be used if explicitly specified.

Furthermore, regarding the new response property reasoning_tokens, since it depends on ChatCompletionChunk from openai.types.chat, I am not including it in the current scope of work. However, as the reasoning result chunks are preserved as inner_content, this value is expected to be included as is in the current implementation.

[moonbox3]

Hi @ymuichiro thanks for the follow up details. Just getting back to the office after holidays - apologies for the slow reply.

As I have a lot of items to go through, let me dig deeper into this today, and I can provide some more comments. As per @eavanvalkenburg's comment above, we do need to think about how the chat history will be handled now that o1 doesn't support the system message -- today one can specify the system message via the constructor, so we'd need to now handle this in a different way.

Another quick thought is: could we have OpenAIReasoningPromptExecutionSettings, which would contain the new properties? That way we're not trying to add too much to the chat prompt execution settings -- nothing is needing to change with the chat models. I also kind of like OpenAIReasoning instead of OpenAIChatReasoning but that might be a personal preference.

Let me circle back soon. And thank you again for all your work on this thus far.

[eavanvalkenburg]

I'm not sure if a seperate PES makes things easier or harder, not having too many settings is great, but we rely on the class property for the "right" PES at different points and having two possibly valid PES's means headaches for that, we could say, check the model id, if starts with o then use Reasoning, but that doesn't work for Azure OpenAI since there we can't reason about which model is actually behind a deployment, so I don't think that would be ideal, I think keeping it simple and add the new params with None defaults, is still easier.

[eavanvalkenburg]

@ymuichiro thanks for your patience, me and @moonbox3 had a chat on this, the current PR is good, and please also add the two new params (max_completion_tokens and reasoning_effort) to the existing OpenAIChatPromptExecutionSettings, we will need to have a broader discussion in our team on how to handle this more broadly so for now we will just add without breaking anyone! Thanks again!

[ymuichiro]

@moonbox3 @eavanvalkenburg

Thank you very much for your review. I have just added a commit with additional modifications.
Regarding the changes, I thought splitting the commits would make them too granular, so I have rebased them into a single commit o1 Provisional Support.

[eavanvalkenburg]

Don't worry about your commits, the pr gets squashed on merge anyway

[moonbox3]

Don't worry about your commits, the pr gets squashed on merge anyway

Was just going to write this -- you beat me to it!

[markwallace-microsoft]

Python Test Coverage Report •

File	Stmts	Miss	Cover	Missing
semantic_kernel/connectors/ai/open_ai/prompt_execution_settings
open_ai_prompt_execution_settings.py	99	1	99%	139
semantic_kernel/contents
chat_history.py	165	2	99%	109, 114
TOTAL	16759	1771	89%

Python Unit Test Overview

Tests	Skipped	Failures	Errors	Time
3000	4 💤	1 ❌	0 🔥	1m 11s ⏱️

[moonbox3]

Thanks for your help on this, @ymuichiro.

[moonbox3]

My only other suggestion for this (along with fixing the unit tests), is that we should add a concept sample to show how to leverage the o1 model -- and make lots of comments around what is/ what is not supported:

• developer message instead of system
• parallel tool calls not yet supported (should set PES parallel_tool_calls = False
• unsupported API parameters: temperature, top_p, presence_penalty, frequency_penalty, logprobs, top_logprobs, logit_bias

[ymuichiro]

Understood. I will add a note in the comments indicating that this is a property specific to o1. Additionally, I will include examples of how to use o1 in directories like /python/samples/concepts/reasoning/**.

[ymuichiro]

To resolve the test error, I investigated the cause and found a challenging issue, so I am documenting it for now. The error occurs in the Azure AI Inference section. The test verifies whether all defined roles can be parsed by the Message class of Azure AI Inference.

semantic-kernel/python/semantic_kernel/connectors/ai/azure_ai_inference/services/utils.py

Line 138 in 2a5e51b

MESSAGE_CONVERTERS: dict[AuthorRole, Callable[[ChatMessageContent], ChatRequestMessage]] = {

To pass the test, we would need to link the AuthorRole defined on the Semantic Kernel side with the ChatRole defined in the Azure AI Inference SDK. However, the Azure AI Inference SDK does not yet include the DEVELOPER role...

https://github.com/Azure/azure-sdk-for-python/blob/3fef975949b7fcb0a45fdde599902d3e39a6c0d0/sdk/ai/azure-ai-inference/azure/ai/inference/models/_enums.py#L29

As a temporary measure, we could prepare a placeholder function with a TODO: note or similar. However, this approach risks missing future updates, so I have started considering how best to address this issue ☹️

[eavanvalkenburg]

@ymuichiro we had a chat on this one, we think it's best if the other classes raise an error for now when presented with a DEVELOPER message, that way we don't have to introduce behavior changes later on (the alternative is to map from developer to system, but then in the future if Azure AI Inference does support Developer it will change the behavior)