How to customize response handlers? #6198
-
Problems I encounteredWhen I use
Because the LLM what I used return data like: data: {"id":"d64e9540-d213-4907-954e-cfc93a840029","choices":[{"delta":{"content":" today","function_call":null,"role":null,"tool_calls":null},"finish_reason":null,"index":0,"logprobs":null}],"created":6030235,"model":"deepseek-chat","object":"chat.completion.chunk","system_fingerprint":null,"usage":null}
data: {"id":"d64e9540-d213-4907-954e-cfc93a840029","choices":[{"delta":{"content":"?","function_call":null,"role":null,"tool_calls":null},"finish_reason":null,"index":0,"logprobs":null}],"created":6030235,"model":"deepseek-chat","object":"chat.completion.chunk","system_fingerprint":null,"usage":null}
data: {"id":"d64e9540-d213-4907-954e-cfc93a840029","choices":[{"delta":{"content":"","function_call":null,"role":null,"tool_calls":null},"finish_reason":"stop","index":0,"logprobs":null}],"created":6030235,"model":"DeepSeek-LLM-67B-chat","object":"chat.completion.chunk","system_fingerprint":null,"usage":{"prompt_tokens":16,"completion_tokens":10,"total_tokens":26}}
data: [DONE]
All the previous data can be obtained. Please pay attention to the end mark ( This is not a json format. The parsing may return null and an error may be reported when using it later. SolutionIs it possible to customize the parsing process so that I can perform special processing on the data returned by different language models and finally process it into a unified format? |
Beta Was this translation helpful? Give feedback.
Replies: 1 comment 2 replies
-
@niltor When you configure AI connector, it's possible to pass custom |
Beta Was this translation helpful? Give feedback.
@niltor Here is an example how to inject custom
HttpClient
in connector:https://github.com/microsoft/semantic-kernel/blob/2530367c17c71c6210afc4b890f97847ac557928/dotnet/samples/Concepts/ChatCompletion/Connectors_CustomHttpClient.cs
It's still will be possible to use
InvokePromptStreamingAsync
method. We don't have examples for such scenario yet, but we do have unit tests that you can use as a reference:semantic-kernel/dotnet/src/Connectors/Connectors.UnitTests/OpenAI/ChatCompletion/AzureOpenAIChatCompletionServiceTests.cs
Lines 432 to 451 in 2530367