Starting from Filters to talk about invoke and ChatCompletionService

[guru98]

Filters rapresents a great value for the framework and I appreciate the effort to extend to auto functions agens and so on. Still I think that there is an issue when. for example you use ChatCompletionServices directly instead use Kernel.Invoke. In my mind the framework should have a preferred way to use it and should have all necessary feature to use it at the best (e.g. history management). Call directly a service or a specific function could be possible but not recomended as you loose consistency. From this point of view all the examples that use ChatCompletitionServices, for example, could generate confusion about if filters are available in those scenario and when to use invoke instead ChatCompletionService

[dmytrostruk]

@guru98 Thank you very much for feedback!
The problem with applying filters on ChatCompletionService level is that we can do that for chat completion services that exist today (e.g. OpenAI, Google, Hugging Face), but it's also possible to inject your custom chat completion service. In this case, you will have to apply filtering logic manually, while if you invoke chat completion through kernel, filters will be invoked automatically, no matter which actual chat completion service you use.

It's still possible to achieve similar behavior, but we will have to re-think ChatCompletionService abstraction (e.g. use abstract class and provide some default implementation that will trigger filters), but we should think if this is really the problem we want to resolve.

Maybe we should understand why there is a need to call ChatCompletionService directly instead of Kernel.InvokeAsync, for example if it's for chat history management, then maybe we should improve it from this perspective.

Also, current filters provide a context with information like KernelFunction, KernelArguments, FunctionResult etc., so it is specific to kernel/function invocation. While in chat completion services we operate with different type of data - LLM-related settings, text content, audio content, etc. So, even if we make a decision to apply filtering on chat completion service level, there should be new type of filters added which will receive information specific to chat completion service.

[guru98]

Thanks Dmytro, i 100% understand your point but in this case my view is that use of invoke should be pushed and clearly preferred in almost totality of the case. At the end Semantic Kernel provide tools, service tha simplify and hopefully make an abstraction indipendenlty from model. So my feedback is that you should reduce the number of case where is necessary use ChatCompletionService instead kernel.invoke. For example where is better to use ChatCompletionServices directly. For sure it have a better management of history but, probably, you could try to provide it at invoke level

…

________________________________ Da: Dmytro Struk ***@***.***> Inviato: venerdì 10 maggio 2024 17:03 A: microsoft/semantic-kernel ***@***.***> Cc: Ugo Castellani ***@***.***>; Mention ***@***.***> Oggetto: Re: [microsoft/semantic-kernel] Starting from Filters to talk about invoke and ChatCompletionService (Issue #6181) @guru98<https://github.com/guru98> Thank you very much for feedback! The problem with applying filters on ChatCompletionService level is that we can do that for chat completion services that exist today (e.g. OpenAI, Google, Hugging Face), but it's also possible to inject your custom chat completion service. In this case, you will have to apply filtering logic manually, while if you invoke chat completion through kernel, filters will be invoked automatically, no matter which actual chat completion service you use. It's still possible to achieve similar behavior, but we will have to re-think ChatCompletionService abstraction (e.g. use abstract class and provide some default implementation that will trigger filters), but we should think if this is really the problem we want to resolve. Maybe we should understand why there is a need to call ChatCompletionService directly instead of Kernel.InvokeAsync, for example if it's for chat history management, then maybe we should improve it from this perspective. Also, current filters provide a context with information like KernelFunction, KernelArguments, FunctionResult etc., so it is specific to kernel/function invocation. While in chat completion services we operate with different type of data - LLM-related settings, text content, audio content, etc. So, even if we make a decision to apply filtering on chat completion service level, there should be new type of filters added which will receive information specific to chat completion service. — Reply to this email directly, view it on GitHub<#6181 (comment)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AA77TTWXCMXXDRI6WH42VMDZBTOVBAVCNFSM6AAAAABHQYDG6WVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDCMBUG43DSMRXGY>. You are receiving this because you were mentioned.Message ID: ***@***.***>

[dmytrostruk]

@guru98 That makes sense and thanks a lot for this feedback, it's really helpful! I agree, we should minimize the need to use chat completion service directly. There are other benefits of using kernel instead of chat completion service such as telemetry, prompt templating etc.

[guru98]

agree my question is when i should use Chat completion vs kernel?

[dmytrostruk]

@guru98 You should prefer kernel over chat completion in most cases. As mentioned above, chat completion is useful when you want to have more control over chat history or manual function calling, but I guess this is the only case. In all other cases, kernel should be a preference.

Also, it's worth to mention, that if you register filters, use chat completion service directly, pass kernel and enable automatic function calling, then the filters will be invoked, because in this scenario there is function invocation in place. But you need to pass kernel to chat completion service invocation.

It's also possible to inject custom HttpClient, in case you want more observability for each chat completion request/response.

Please let me know if that answers your question. Thanks!

[guru98]

Yes thanks. . Inviato da Outlook per Android<https://aka.ms/AAb9ysg>

…

________________________________ From: Dmytro Struk ***@***.***> Sent: Friday, May 10, 2024 6:13:05 PM To: microsoft/semantic-kernel ***@***.***> Cc: Ugo Castellani ***@***.***>; Mention ***@***.***> Subject: Re: [microsoft/semantic-kernel] Starting from Filters to talk about invoke and ChatCompletionService (Discussion #6185) @guru98<https://github.com/guru98> You should prefer kernel over chat completion in most cases. As mentioned above, chat completion is useful when you want to have more control over chat history or manual function calling, but I guess this is the only case. In all other cases, kernel should be a preference. Also, it's worth to mention, that if you register filters, use chat completion service directly, pass kernel and enable automatic function calling, then the filters will be invoked, because in this scenario there is function invocation in place. But you need to pass kernel to chat completion service invocation. It's also possible to inject custom HttpClient, in case you want more observability for each chat completion request/response. Please let me know if that answers your question. Thanks! — Reply to this email directly, view it on GitHub<#6185 (reply in thread)>, or unsubscribe<https://github.com/notifications/unsubscribe-auth/AA77TTQXCWUMHAIXJC3A3ALZBTWZDAVCNFSM6AAAAABHQ4KF2WVHI2DSMVQWIX3LMV43SRDJONRXK43TNFXW4Q3PNVWWK3TUHM4TGOBWGUYDC>. You are receiving this because you were mentioned.Message ID: ***@***.***>