Integration with Semantic Kernel, Guidance and Prompt Flow

[surjray]

Autogen is an amazing project. Considering the other LLM related developments in other parts of Microsoft, namely Semantic Kernel, the Guidance LLM flow/prompt templating language and Prompt Flow, how shall these projects or parts thereof be integrated into autogen to ensure that we do not reinvent the wheel ?

Semantic Kernel: https://github.com/microsoft/semantic-kernel
Guidance: https://github.com/guidance-ai/guidance
PromptFlow: https://github.com/microsoft/promptflow

[Ryangr0]

Interested in this as well. Somehow hooking this into SK would make for a lot of potential.

[sonichi]

Happy to hear suggestions of integration ideas in this thread.

[michaelgpt]

@sonichi I think folks are looking to stakeholders like yourself for guidance (pun intended). Semantic Kernel has a default template engine (basic implementation), Promptflow leverages jinja, and then you have Guidance...

When it comes to Promptflow + SK, I see PF's value being the ability to create prompt chains (orchestration), but couldn't you delegate orchestration to any SK Planners that are implemented (Stepwise, etc.) albeit at the cost of determinism...?

The other value Promptflow provides is the test/eval capabilities. How does that play alongside semantic kernel? How can I take my existing SK plugins (aka skills) and leverage PF to create test/eval loops without having to fully port everything into a flow. I'm also curious what best practices are around SK semantic functions which use SK's default template engine and PF jinja...? Doesn't seem like a good idea to use both. So would you just define python functions within your Flow that invoke SK plugins? If that is the case, what value is PF actually delivering? I can orchestrate (compose) SK plugin calls solely using SK...

These are just some the thoughts I have off the cusp. Interested in hearing what folks think and what current approaches seem to be the most promising that are being tinkered with 😄

[sonichi]

I'm certainly interested in this question. But I'll need to be heads down for solving #203 in the near term and won't have time to dive deep. Please continue the discussion.

[HusainMkd]

Hi @sonichi - has there been any update on this issue? I'm also really keen on guidance about how to use promptflow with Autogen, or if I need to choose between them separately. Many thanks!

[sonichi]

I've seen some integrations of autogen into SK, and a video about using Guidance + AutoGen. For using PF with autogen, could you share the tentative use case?

[Rocky0102]

Autogen/Prompflow/SemanticKernel all seem to enable better LLM apps, but why not all in one project? Because they are created by different departments?

[Karl-AA]

@sonichi For me, idea of integrating PF and Autogen would be for evaluation purposes (and deployment too, via PF?)

PF evaluation looks really great, and seems to have a solid framework for CI/CD too.

[sonichi]

Could you share more how you envision PF to be integrated with AutoGen for evaluation?
cc @julianakiseleva @afourney

[Karl-AA]

I'm not 100% comfortable with Autogen yet so it is a little difficult to envision.

With that said, the main aspects that I care about in terms of integrating with PF evaluation is the ability to run batches of simulated interactions, collect results, and compare to the "ideal" answer for each interaction, contributing to quantitative scoring using some metrics.

How to fit Autogen into PF is the unclear part for me - but the ability to do this kind of evaluation is really valuable, if not necessary, for iterating on the design and getting solutions ready for production.

Making your own eval studio with these capabilities could be another option right, because it shouldn't be super complicated to build out the framework for batch testing, answer comparison, metric comparison, etc? I say this because the new UI is pretty awesome, and it could be an add-on for that perhaps...

[L-Sypniewski]

Is there any official documentation clarifying when to use Semantic Kernel vs. AutoGen?

It looks like they overlap in many aspects, so it would be good to know when to use which solution.

I've seen 'AutoGen Agents Meet Semantic Kernel' blog post, but it's looks like Semantic Kernel got new features since it was posted.

[sonichi]

I'm not 100% comfortable with Autogen yet so it is a little difficult to envision.

With that said, the main aspects that I care about in terms of integrating with PF evaluation is the ability to run batches of simulated interactions, collect results, and compare to the "ideal" answer for each interaction, contributing to quantitative scoring using some metrics.

How to fit Autogen into PF is the unclear part for me - but the ability to do this kind of evaluation is really valuable, if not necessary, for iterating on the design and getting solutions ready for production.

Making your own eval studio with these capabilities could be another option right, because it shouldn't be super complicated to build out the framework for batch testing, answer comparison, metric comparison, etc? I say this because the new UI is pretty awesome, and it could be an add-on for that perhaps...

Thanks for the suggestion. Currently we have some tools for evaluation:
AgentEval
Testbed

cc @julianakiseleva @afourney @qingyun-wu @victordibia

[NamanJain-Nash]

"I'm interested in exploring the integration of semantic kernels as agents in autogen . Currently, we have various model-based agents, and I'm wondering if we can incorporate semantic kernels into the mix. The idea is to leverage their plugins, planners, and personas.

As a case study, imagine having 2-3 semantic kernels that function as agents, each with different personas and tools. Through the use of auto-gen, these agents could communicate with each other to collaboratively solve a problem. This approach could be particularly beneficial when dealing with issues such as retrieving accurate information and managing toolkits.

The goal here is to address situations where relying on a single agent with numerous tools might pose challenges, be it security concerns or environmental limitations. By using multiple semantic kernels with distinct personas, we can enhance control and tackle various use cases more effectively."

As to my current understanding, the semantic kernel is more of a single-agent type system and Auto Gen is a multi-agent system at least by initial thoughts so why not make a single agent be used in a multi-agent system like in general it just sounds like a next step [Note I might be oversimplifying the thought process of both the projects and might be going in the wrong direction.]

[d3ssy]

I'm interested din this discussion too. Specifically in what the best pattern is to leverage Autogen within Prompt Flow without ending up putting black boxes inside other black boxes and lose traceability.

@NamanJain-Nash if you already have model based agents, wouldn't they already be performing all of the functionality that SK offers since, by definition, agents already have defined "personas" and tools they can access. Can't you just use more Autogen agents with well defined capabilities and usage patterns for your use case?

[chengyuliu-msft]

Following this topic, and I am not sure how AutoGen solutions lifecycle can be managed by Prompt Flow or other tools ?

I have seen some solutions where SK and Prompt Flow are integrated to manage life cycle of gen AIs by utilizing AML. Prompt flow provides a way to author and systematically evaluate the (chat and completion scenarios), and deploy the solution to Azure ML.

[sonichi]

@surjray @michaelgpt @HusainMkd @Rocky0102 @Karl-AA @d3ssy @chengyuliu-msft Thanks for your interest in the integration with PF. Could you review #2369 ?

[johnnieskywalker]

Hi I'm bit confused whether it's good idea to use both autogen and PromptFlow or just pick one of tools for LLM app development. There's a llmops repo that provides information on how to version prompts and flows in the repo and and be deployed with github actions as well. In promptflow one can call python funcions as well in a runtime. Autogen seems to be a framework for controling agents, terminate their responses, define functions and so on. Can you share some example or article on how these 2 tools should or shouldn't be used together? @krishnashed

[ekzhu]

@johnnieskywalker example on promptflow integration: https://github.com/microsoft/autogen/tree/main/samples/apps/promptflow-autogen