Unit (or maybe integration?) testing of semantic functions

[lukasz-appstream]

Hi everyone,

I wanted to initiate a discussion on the topic of testing the prompts we use as skills. I have been feeling quite uneasy about the possibility of switching between model versions, updates to the model, or even my own fine-tuning potentially leading to changes in completions that could break my application.

What's even worse is that these changes can modify the behavior of my software without any visible issues.

To address this concern, I have come up with a couple of ideas for testing our prompts:

1. Introduce a testcases.txt or testcases.json file that contains a set of predefined test cases (similar to examples). We can then execute these tests by starting the application with the Semantic Kernel using the command "myapp.exe test". This way, we can ensure that our prompts produce the expected results consistently.

2. Develop a lightweight SemanticTest or SemanticKernel.Test library specifically designed for testing semantic skills. This library can provide a framework for creating and running tests to verify the behavior and stability of our skills.

I would love to hear your thoughts on these ideas and if you have any additional suggestions for improving our testing process.

Keep up the amazing work, everyone!

Best regards,
Lukasz

[ghost]

I sent you a note in Linked in. I would be interested in seeing what you have in mind. I'm currently building a sample CoPilot solution in Asp.net and am using MOQ over most of semantic kernel. I just started looking at refactoring the sample solution to be more testable in unit test scenarios.

[evchaki]

@lukasz-appstream - check out Prompt Flow for this - https://learn.microsoft.com/en-us/azure/machine-learning/prompt-flow/get-started-prompt-flow?view=azureml-api-2