From b715223c3d03b17a27b67cd58eba355a2fe5e80a Mon Sep 17 00:00:00 2001 From: siddharth Sambharia Date: Wed, 21 Aug 2024 15:48:11 +0530 Subject: [PATCH 01/14] Portkey Integration with Autogen --- website/docs/ecosystem/portkey.md | 210 ++++++++++++++++++++++++++++++ 1 file changed, 210 insertions(+) create mode 100644 website/docs/ecosystem/portkey.md diff --git a/website/docs/ecosystem/portkey.md b/website/docs/ecosystem/portkey.md new file mode 100644 index 000000000000..34d6228c5001 --- /dev/null +++ b/website/docs/ecosystem/portkey.md @@ -0,0 +1,210 @@ +# Portkey Integration with Autogen + Portkey Metrics Visualization + +[Portkey](https://portkey.ai) is a 2-line upgrade to make your Autogen agents reliable, cost-efficient, and fast. + +Portkey adds 4 core production capabilities to any Autogen agent: +1. Routing to 200+ LLMs +2. Making each LLM call more robust +3. Full-stack tracing & cost, performance analytics +4. Real-time guardrails to enforce behavior + +## Getting Started + +1. **Install Required Packages:** + + ```bash + pip install -qU pyautogen portkey-ai + ``` + +2. **Configure Autogen with Portkey:** + + ```python + from autogen import AssistantAgent, UserProxyAgent, config_list_from_json + from portkey_ai import PORTKEY_GATEWAY_URL, createHeaders + + config = [ + { + "api_key": "OPENAI_API_KEY", + "model": "gpt-3.5-turbo", + "base_url": PORTKEY_GATEWAY_URL, + "api_type": "openai", + "default_headers": createHeaders( + api_key="YOUR_PORTKEY_API_KEY", + provider="openai", + ) + } + ] + ``` + + Generate your API key in the [Portkey Dashboard](https://app.portkey.ai/). + +And, that's it! With just this, you can start logging all of your Autogen requests and make them reliable. + +3. **Let's Run your Agent** + +``` python +import autogen + +# Create user proxy agent, coder, product manager + + +user_proxy = autogen.UserProxyAgent( + name="User_proxy", + system_message="A human admin who will give the idea and run the code provided by Coder.", + code_execution_config={"last_n_messages": 2, "work_dir": "groupchat"}, + human_input_mode="ALWAYS", +) + + +coder = autogen.AssistantAgent( + name="Coder", + system_message = "You are a Python developer who is good at developing games. You work with Product Manager." + llm_config={"config_list": config}, +) + +# Create groupchat +groupchat = autogen.GroupChat( + agents=[user_proxy, coder], messages=[]) +manager = autogen.GroupChatManager(groupchat=groupchat, llm_config={"config_list": config}) + + + +# Start the conversation +user_proxy.initiate_chat( + manager, message="Build a classic & basic pong game with 2 players in python") + +``` +
+Here’s the output from your Agent’s run on Portkey's dashboard
+Portkey Dashboard + + + + + + +## Key Features +Portkey offers a range of advanced features to enhance your Autogen agents. Here’s an overview + +| Feature | Description | +|---------|-------------| +| 🌐 [Multi-LLM Integration](#interoperability) | Access 200+ LLMs with simple configuration changes | +| 🛡️ [Enhanced Reliability](#reliability) | Implement fallbacks, load balancing, retries, and much more | +| 📊 [Advanced Metrics](#metrics) | Track costs, tokens, latency, and 40+ custom metrics effortlessly | +| 🔍 [Detailed Traces and Logs](#comprehensive-logging) | Gain insights into every agent action and decision | +| 🚧 [Guardrails](#guardrails) | Enforce agent behavior with real-time checks on inputs and outputs | +| 🔄 [Continuous Optimization](#continuous-improvement) | Capture user feedback for ongoing agent improvements | +| 💾 [Smart Caching](#caching) | Reduce costs and latency with built-in caching mechanisms | +| 🔐 [Enterprise-Grade Security](#security-and-compliance) | Set budget limits and implement fine-grained access controls | + + +## Colab Notebook + +For a hands-on example of integrating Portkey with Autogen, check out our notebook

[![Google Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://git.new/Portkey-Autogen) . + + + +## Advanced Features + +### Interoperability + +Easily switch between **200+ LLMs** by changing the `provider` and API key in your configuration. + +#### Example: Switching from OpenAI to Azure OpenAI + +```python +config = [ + { + "api_key": "api-key", + "model": "gpt-3.5-turbo", + "base_url": PORTKEY_GATEWAY_URL, + "api_type": "openai", + "default_headers": createHeaders( + api_key="YOUR_PORTKEY_API_KEY", + provider="azure-openai", + virtual_key="AZURE_VIRTUAL_KEY" + ) + } +] +``` + +### Reliability + +Implement fallbacks, load balancing, and automatic retries to make your agents more resilient. + +```python +portkey_config = { + "retry": { + "attempts": 5 + }, + "strategy": { + "mode": "loadbalance" # Options: "loadbalance" or "fallback" + }, + "targets": [ + { + "provider": "openai", + "api_key": "OpenAI_API_Key" + }, + { + "provider": "anthropic", + "api_key": "Anthropic_API_Key" + } + ] +} +``` + +### Metrics + +Agent runs are complex. Portkey automatically logs **40+ comprehensive metrics** for your AI agents, including cost, tokens used, latency, etc. Whether you need a broad overview or granular insights into your agent runs, Portkey's customizable filters provide the metrics you need. + +
+ Portkey's Observability Dashboard +Portkey Dashboard +
+ +### Comprehensive Logging + +Access detailed logs and traces of agent activities, function calls, and errors. Filter logs based on multiple parameters for in-depth analysis. + +
+ Traces + Portkey Logging Interface +
+ +
+ Logs + Portkey Metrics Visualization +
+ +### Guardrails +Autogen agents, while powerful, can sometimes produce unexpected or undesired outputs. Portkey's Guardrails feature helps enforce agent behavior in real-time, ensuring your Autogen agents operate within specified parameters. Verify both the **inputs** to and *outputs* from your agents to ensure they adhere to specified formats and content guidelines. Learn more about Portkey's Guardrails [here](https://docs.portkey.ai/product/guardrails) + +### Continuous Improvement + +Capture qualitative and quantitative user feedback on your requests to continuously enhance your agent performance. + +### Caching + +Reduce costs and latency with Portkey's built-in caching system. + +```python +portkey_config = { + "cache": { + "mode": "semantic" # Options: "simple" or "semantic" + } +} +``` + +### Security and Compliance + +Set budget limits on provider API keys and implement fine-grained user roles and permissions for both your application and the Portkey APIs. + +## Additional Resources + +- [📘 Portkey Documentation](https://docs.portkey.ai) +- [🐦 Twitter](https://twitter.com/portkeyai) +- [💬 Discord Community](https://discord.gg/JHPt4C7r) +- [📊 Portkey App](https://app.portkey.ai) + +For more information on using these features and setting up your Config, please refer to the [Portkey documentation](https://docs.portkey.ai). From 919e1ecffac5558e3b5245f6ebee5e02b0646126 Mon Sep 17 00:00:00 2001 From: siddharth Sambharia Date: Sun, 25 Aug 2024 12:44:00 +0530 Subject: [PATCH 02/14] Update website/docs/ecosystem/portkey.md Co-authored-by: gagb --- website/docs/ecosystem/portkey.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/ecosystem/portkey.md b/website/docs/ecosystem/portkey.md index 34d6228c5001..53e5612a612f 100644 --- a/website/docs/ecosystem/portkey.md +++ b/website/docs/ecosystem/portkey.md @@ -3,7 +3,7 @@ [Portkey](https://portkey.ai) is a 2-line upgrade to make your Autogen agents reliable, cost-efficient, and fast. -Portkey adds 4 core production capabilities to any Autogen agent: +Portkey adds 4 core production capabilities to any AutoGen agent: 1. Routing to 200+ LLMs 2. Making each LLM call more robust 3. Full-stack tracing & cost, performance analytics From 104b8095691957c755197070b6b73dc5ee2275ae Mon Sep 17 00:00:00 2001 From: siddharth Sambharia Date: Sun, 25 Aug 2024 12:44:20 +0530 Subject: [PATCH 03/14] Update website/docs/ecosystem/portkey.md Co-authored-by: gagb --- website/docs/ecosystem/portkey.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/ecosystem/portkey.md b/website/docs/ecosystem/portkey.md index 53e5612a612f..842be5fd27ce 100644 --- a/website/docs/ecosystem/portkey.md +++ b/website/docs/ecosystem/portkey.md @@ -1,7 +1,7 @@ # Portkey Integration with Autogen Portkey Metrics Visualization -[Portkey](https://portkey.ai) is a 2-line upgrade to make your Autogen agents reliable, cost-efficient, and fast. +[Portkey](https://portkey.ai) is a 2-line upgrade to make your AutoGen agents reliable, cost-efficient, and fast. Portkey adds 4 core production capabilities to any AutoGen agent: 1. Routing to 200+ LLMs From f1a21fead25fe51e350c85f793d53eb9437b06e3 Mon Sep 17 00:00:00 2001 From: siddharth Sambharia Date: Sun, 25 Aug 2024 12:44:39 +0530 Subject: [PATCH 04/14] Update website/docs/ecosystem/portkey.md Co-authored-by: gagb --- website/docs/ecosystem/portkey.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/ecosystem/portkey.md b/website/docs/ecosystem/portkey.md index 842be5fd27ce..077899a124cb 100644 --- a/website/docs/ecosystem/portkey.md +++ b/website/docs/ecosystem/portkey.md @@ -1,4 +1,4 @@ -# Portkey Integration with Autogen +# Portkey Integration with AutoGen Portkey Metrics Visualization [Portkey](https://portkey.ai) is a 2-line upgrade to make your AutoGen agents reliable, cost-efficient, and fast. From 4417cf8e9d2e53f64586a8bafdff6ff61cea34bb Mon Sep 17 00:00:00 2001 From: gagb Date: Mon, 26 Aug 2024 00:02:32 -0700 Subject: [PATCH 05/14] Update website/docs/ecosystem/portkey.md --- website/docs/ecosystem/portkey.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/ecosystem/portkey.md b/website/docs/ecosystem/portkey.md index 077899a124cb..b6e3300b1c19 100644 --- a/website/docs/ecosystem/portkey.md +++ b/website/docs/ecosystem/portkey.md @@ -39,7 +39,7 @@ Portkey adds 4 core production capabilities to any AutoGen agent: Generate your API key in the [Portkey Dashboard](https://app.portkey.ai/). -And, that's it! With just this, you can start logging all of your Autogen requests and make them reliable. +And, that's it! With just this, you can start logging all of your AutoGen requests and make them reliable. 3. **Let's Run your Agent** From 4bfff479da547162b3d5b2b6e8b3d3e60e9b3afb Mon Sep 17 00:00:00 2001 From: gagb Date: Mon, 26 Aug 2024 00:05:49 -0700 Subject: [PATCH 06/14] Update website/docs/ecosystem/portkey.md --- website/docs/ecosystem/portkey.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/ecosystem/portkey.md b/website/docs/ecosystem/portkey.md index b6e3300b1c19..5620236624d9 100644 --- a/website/docs/ecosystem/portkey.md +++ b/website/docs/ecosystem/portkey.md @@ -17,7 +17,7 @@ Portkey adds 4 core production capabilities to any AutoGen agent: pip install -qU pyautogen portkey-ai ``` -2. **Configure Autogen with Portkey:** +2. **Configure AutoGen with Portkey:** ```python from autogen import AssistantAgent, UserProxyAgent, config_list_from_json From a63517c837a96b4c3c3361d226ca96c7a0ef137f Mon Sep 17 00:00:00 2001 From: gagb Date: Mon, 26 Aug 2024 00:05:57 -0700 Subject: [PATCH 07/14] Update website/docs/ecosystem/portkey.md --- website/docs/ecosystem/portkey.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/ecosystem/portkey.md b/website/docs/ecosystem/portkey.md index 5620236624d9..d994929b4afb 100644 --- a/website/docs/ecosystem/portkey.md +++ b/website/docs/ecosystem/portkey.md @@ -85,7 +85,7 @@ Here’s the output from your Agent’s run on Portkey's dashboard
## Key Features -Portkey offers a range of advanced features to enhance your Autogen agents. Here’s an overview +Portkey offers a range of advanced features to enhance your AutoGen agents. Here’s an overview | Feature | Description | |---------|-------------| From cab3c5442b29707a1ba49d79e8c2ebd5c98d3568 Mon Sep 17 00:00:00 2001 From: gagb Date: Mon, 26 Aug 2024 00:06:07 -0700 Subject: [PATCH 08/14] Update website/docs/ecosystem/portkey.md --- website/docs/ecosystem/portkey.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/ecosystem/portkey.md b/website/docs/ecosystem/portkey.md index d994929b4afb..a0b66c3683cb 100644 --- a/website/docs/ecosystem/portkey.md +++ b/website/docs/ecosystem/portkey.md @@ -178,7 +178,7 @@ Access detailed logs and traces of agent activities, function calls, and errors. ### Guardrails -Autogen agents, while powerful, can sometimes produce unexpected or undesired outputs. Portkey's Guardrails feature helps enforce agent behavior in real-time, ensuring your Autogen agents operate within specified parameters. Verify both the **inputs** to and *outputs* from your agents to ensure they adhere to specified formats and content guidelines. Learn more about Portkey's Guardrails [here](https://docs.portkey.ai/product/guardrails) +Autogen agents, while powerful, can sometimes produce unexpected or undesired outputs. Portkey's Guardrails feature helps enforce agent behavior in real-time, ensuring your AutoGen agents operate within specified parameters. Verify both the **inputs** to and *outputs* from your agents to ensure they adhere to specified formats and content guidelines. Learn more about Portkey's Guardrails [here](https://docs.portkey.ai/product/guardrails) ### Continuous Improvement From 1fd287a134b27233e17e135dd5b45e6fc1b48fe4 Mon Sep 17 00:00:00 2001 From: gagb Date: Mon, 26 Aug 2024 00:08:29 -0700 Subject: [PATCH 09/14] Update website/docs/ecosystem/portkey.md --- website/docs/ecosystem/portkey.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/ecosystem/portkey.md b/website/docs/ecosystem/portkey.md index a0b66c3683cb..085e72958e9d 100644 --- a/website/docs/ecosystem/portkey.md +++ b/website/docs/ecosystem/portkey.md @@ -178,7 +178,7 @@ Access detailed logs and traces of agent activities, function calls, and errors. ### Guardrails -Autogen agents, while powerful, can sometimes produce unexpected or undesired outputs. Portkey's Guardrails feature helps enforce agent behavior in real-time, ensuring your AutoGen agents operate within specified parameters. Verify both the **inputs** to and *outputs* from your agents to ensure they adhere to specified formats and content guidelines. Learn more about Portkey's Guardrails [here](https://docs.portkey.ai/product/guardrails) +AutoGen agents, while powerful, can sometimes produce unexpected or undesired outputs. Portkey's Guardrails feature helps enforce agent behavior in real-time, ensuring your AutoGen agents operate within specified parameters. Verify both the **inputs** to and *outputs* from your agents to ensure they adhere to specified formats and content guidelines. Learn more about Portkey's Guardrails [here](https://docs.portkey.ai/product/guardrails) ### Continuous Improvement From 2f8900925c69027119943a3f904efd9353befe67 Mon Sep 17 00:00:00 2001 From: siddharth Sambharia Date: Mon, 26 Aug 2024 12:56:21 +0530 Subject: [PATCH 10/14] Update portkey.md with changes --- website/docs/ecosystem/portkey.md | 22 ++++++++++++++-------- 1 file changed, 14 insertions(+), 8 deletions(-) diff --git a/website/docs/ecosystem/portkey.md b/website/docs/ecosystem/portkey.md index 085e72958e9d..e762a18d307f 100644 --- a/website/docs/ecosystem/portkey.md +++ b/website/docs/ecosystem/portkey.md @@ -134,26 +134,32 @@ config = [ Implement fallbacks, load balancing, and automatic retries to make your agents more resilient. ```python -portkey_config = { - "retry": { - "attempts": 5 - }, +{ "strategy": { - "mode": "loadbalance" # Options: "loadbalance" or "fallback" + "mode": "fallback" # Options: "loadbalance" or "fallback" }, "targets": [ { "provider": "openai", - "api_key": "OpenAI_API_Key" + "api_key": "openai-api-key", + "override_params": { + "top_k": "0.4", + "max_tokens": "100" + } }, { "provider": "anthropic", - "api_key": "Anthropic_API_Key" + "api_key": "anthropic-api-key", + "override_params": { + "top_p": "0.6", + "model": "claude-3-5-sonnet-20240620" + } } ] } ``` - +Learn more about [Portkey Config object here](https://docs.portkey.ai/docs/product/ai-gateway-streamline-llm-integrations/configs). +Be Careful to Load-Balance/Fallback to providers that don't support tool calling when the request contains a function call. ### Metrics Agent runs are complex. Portkey automatically logs **40+ comprehensive metrics** for your AI agents, including cost, tokens used, latency, etc. Whether you need a broad overview or granular insights into your agent runs, Portkey's customizable filters provide the metrics you need. From 7fa62b02e4bcbf4be2f8ad39911def79a6ce8625 Mon Sep 17 00:00:00 2001 From: siddharth Sambharia Date: Tue, 27 Aug 2024 10:54:18 +0530 Subject: [PATCH 11/14] Update website/docs/ecosystem/portkey.md Co-authored-by: Mark Sze <66362098+marklysze@users.noreply.github.com> --- website/docs/ecosystem/portkey.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/website/docs/ecosystem/portkey.md b/website/docs/ecosystem/portkey.md index e762a18d307f..678b409e478e 100644 --- a/website/docs/ecosystem/portkey.md +++ b/website/docs/ecosystem/portkey.md @@ -59,7 +59,7 @@ user_proxy = autogen.UserProxyAgent( coder = autogen.AssistantAgent( name="Coder", - system_message = "You are a Python developer who is good at developing games. You work with Product Manager." + system_message = "You are a Python developer who is good at developing games. You work with Product Manager.", llm_config={"config_list": config}, ) From e77d46df092993c93a067e3fd2b13f372bd077ab Mon Sep 17 00:00:00 2001 From: siddharth Sambharia Date: Thu, 29 Aug 2024 12:39:54 +0530 Subject: [PATCH 12/14] Remove trailing white spaces --- website/docs/ecosystem/portkey.md | 11 ++--------- 1 file changed, 2 insertions(+), 9 deletions(-) diff --git a/website/docs/ecosystem/portkey.md b/website/docs/ecosystem/portkey.md index 678b409e478e..cccb8ca2cb09 100644 --- a/website/docs/ecosystem/portkey.md +++ b/website/docs/ecosystem/portkey.md @@ -12,12 +12,11 @@ Portkey adds 4 core production capabilities to any AutoGen agent: ## Getting Started 1. **Install Required Packages:** - +2. ```bash pip install -qU pyautogen portkey-ai ``` - -2. **Configure AutoGen with Portkey:** +. **Configure AutoGen with Portkey:** ```python from autogen import AssistantAgent, UserProxyAgent, config_list_from_json @@ -73,17 +72,11 @@ manager = autogen.GroupChatManager(groupchat=groupchat, llm_config={"config_list # Start the conversation user_proxy.initiate_chat( manager, message="Build a classic & basic pong game with 2 players in python") - ```
Here’s the output from your Agent’s run on Portkey's dashboard
Portkey Dashboard - - - - - ## Key Features Portkey offers a range of advanced features to enhance your AutoGen agents. Here’s an overview From 170c7c2c05857ca7d9f4b9ad07a97b4b2e25586f Mon Sep 17 00:00:00 2001 From: siddharth Sambharia Date: Fri, 30 Aug 2024 11:36:34 +0530 Subject: [PATCH 13/14] Update portkey.md with mark's note --- website/docs/ecosystem/portkey.md | 1 + 1 file changed, 1 insertion(+) diff --git a/website/docs/ecosystem/portkey.md b/website/docs/ecosystem/portkey.md index cccb8ca2cb09..944941e99868 100644 --- a/website/docs/ecosystem/portkey.md +++ b/website/docs/ecosystem/portkey.md @@ -121,6 +121,7 @@ config = [ } ] ``` +Note: AutoGen messages will go through Portkey's AI Gateway following OpenAI's API signature. Some language models may not work properly because messages need to be in a specific role order. ### Reliability From 28c1ad79d17221f08cd2819beec514a63056a202 Mon Sep 17 00:00:00 2001 From: siddharth Sambharia Date: Sat, 31 Aug 2024 19:16:50 +0530 Subject: [PATCH 14/14] remove trailing white spaces line 15 and empty line at end --- website/docs/ecosystem/portkey.md | 5 ++--- 1 file changed, 2 insertions(+), 3 deletions(-) diff --git a/website/docs/ecosystem/portkey.md b/website/docs/ecosystem/portkey.md index 944941e99868..4825cf78d9a7 100644 --- a/website/docs/ecosystem/portkey.md +++ b/website/docs/ecosystem/portkey.md @@ -12,11 +12,10 @@ Portkey adds 4 core production capabilities to any AutoGen agent: ## Getting Started 1. **Install Required Packages:** -2. - ```bash +2. ```bash pip install -qU pyautogen portkey-ai ``` -. **Configure AutoGen with Portkey:** + **Configure AutoGen with Portkey:** ```python from autogen import AssistantAgent, UserProxyAgent, config_list_from_json