Instantiating Semantic Kernel in multi session AI Web/API Application

[jamesmkfoo23]

I have a question, between the 2 methods below, which should be the better way?

A) For every web request, instantiate kernel, add service, plugins
Setup and teardown would create some overhead.

B) Instantiate kernel, add service and plugins once during app startup.
I would think this is more performant, though perhaps marginal. How

[Maniga]

The better way is to instantiate it at startup, I would say. But sometimes you are not able to do that, like with the feature I need which you can see in this discussion:

Dynamic Creation of chat completion endpoints during runtime #6811

I didn't like the creation of the kernel before each call, so I created a service which handles the creation and holds the semantic kernel instance.

[jamesmkfoo23]

I too have tried to add multiple service/model by service-id to single Kernel. Tried to use adjust plugin prompt config.json parameter to specify service-id, hoping it will call the model based on service-id, but that didn't work.

[dmytrostruk]

@jamesmkfoo23 Here is an example for your scenario for .NET version of Semantic Kernel:

skprompt.txt

Question: {{$input}}

config.json

{
  "schema": 1,
  "description": "Answer any question",
  "execution_settings": {
    "gpt-3.5-turbo": {
      "max_tokens": 4000,
      "temperature": 0.1
    }
  }
}

C#:

var builder = Kernel.CreateBuilder()
    .AddOpenAIChatCompletion(
        modelId: "gpt-3.5-turbo-0613",
        apiKey: Environment.GetEnvironmentVariable("OPENAI_APIKEY"),
        serviceId: "gpt-3.5-turbo")
    .AddOpenAIChatCompletion(
        modelId: "gpt-4-1106-preview",
        apiKey: Environment.GetEnvironmentVariable("OPENAI_APIKEY"),
        serviceId: "gpt-4");

builder.Plugins.AddFromPromptDirectory(pluginDirectory: "MyRepository/Plugins", pluginName: "MyPlugin");

var kernel = builder.Build();

var arguments = new KernelArguments { ["input"] = "Hi, what is the capital of France?" };
var result = await kernel.InvokeAsync(pluginName: "MyPlugin", functionName: "QuestionAnswer", arguments: arguments);

var value = result.GetValue<OpenAIChatMessageContent>()!;

Console.WriteLine(value.Content); // The capital of France is Paris.
Console.WriteLine(value.ModelId); // gpt-3.5-turbo-0613

The key point here is to register multiple chat completion services in Kernel instance, and you can specify in config.json file which service for specific plugin you want to use. This capability allows you to use different models for different plugins.

It's also possible to specify multiple services in your config.json, like here:

semantic-kernel/dotnet/samples/LearnResources/Plugins/Prompts/chat/config.json

Lines 5 to 20 in 148e8de

	"execution_settings": {
	"default": {
	"max_tokens": 1000,
	"temperature": 0
	},
	"gpt-3.5-turbo": {
	"model_id": "gpt-3.5-turbo-0613",
	"max_tokens": 4000,
	"temperature": 0.1
	},
	"gpt-4": {
	"model_id": "gpt-4-1106-preview",
	"max_tokens": 8000,
	"temperature": 0.3
	}
	},

But in this case, it will pick first service in order of registration. If you want to customize this selection logic, you can implement your custom IAIServiceSelector, like in this example:
https://github.com/microsoft/semantic-kernel/blob/main/dotnet/samples/Concepts/Kernel/CustomAIServiceSelector.cs

I hope this will allow you to achieve your scenario. Thanks!

[jamesmkfoo23]

Thank you @dmytrostruk! Hope the config.json service specification works for python SK too :)