You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Copy file name to clipboardExpand all lines: TRANSPARENCY_FAQS.md
+1
Original file line number
Diff line number
Diff line change
@@ -30,6 +30,7 @@ While AutoGen automates LLM workflows, decisions about how to use specific LLM o
30
30
## How was AutoGen evaluated? What metrics are used to measure performance?
31
31
- Current version of AutoGen was evaluated on six applications to illustrate its potential in simplifying the development of high-performance multi-agent applications. These applications are selected based on their real-world relevance, problem difficulty and problem solving capabilities enabled by AutoGen, and innovative potential.
32
32
- These applications involve using AutoGen to solve math problems, question answering, decision making in text world environments, supply chain optimization, etc. For each of these domains AutoGen was evaluated on various success based metrics (i.e., how often the AutoGen based implementation solved the task). And, in some cases, AutoGen based approach was also evaluated on implementation efficiency (e.g., to track reductions in developer effort to build). More details can be found at: https://aka.ms/AutoGen/TechReport
33
+
- The team has conducted tests where a “red” agent attempts to get the default AutoGen assistant to break from its alignment and guardrails. The team has observed that out of 70 attempts to break guardrails, only 1 was successful in producing text that would have been flagged as problematic by Azure OpenAI filters. The team has not observed any evidence that AutoGen (or GPT models as hosted by OpenAI or Azure) can produce novel code exploits or jailbreak prompts, since direct prompts to “be a hacker”, “write exploits”, or “produce a phishing email” are refused by existing filters.
33
34
34
35
## What are the limitations of AutoGen? How can users minimize the impact of AutoGen’s limitations when using the system?
35
36
AutoGen relies on existing LLMs. Experimenting with AutoGen would retain common limitations of large language models; including:
chat_queue (List[Dict]): a list of dictionaries containing the information of the chats.
37
-
Each dictionary should contain the following fields:
37
+
Each dictionary should contain the input arguments for `ConversableAgent.initiate_chat`.
38
+
More specifically, each dictionary could include the following fields:
39
+
recipient: the recipient agent.
40
+
- "sender": the sender agent.
38
41
- "recipient": the recipient agent.
39
-
- "context": any context information, e.g., the request message. The following fields are reserved:
40
-
"message" needs to be provided if the `generate_init_message` method is not overridden.
41
-
Otherwise, input() will be called to get the initial message.
42
-
"summary_method": a string or callable specifying the method to get a summary from the chat. Default is DEFAULT_summary_method, i.e., "last_msg".
43
-
- Supported string are "last_msg" and "reflection_with_llm":
44
-
when set "last_msg", it returns the last message of the dialog as the summary.
45
-
when set "reflection_with_llm", it returns a summary extracted using an llm client.
46
-
`llm_config` must be set in either the recipient or sender.
47
-
"reflection_with_llm" requires the llm_config to be set in either the sender or the recipient.
48
-
- A callable summary_method should take the recipient and sender agent in a chat as input and return a string of summary. E.g,
49
-
```python
50
-
def my_summary_method(
51
-
sender: ConversableAgent,
52
-
recipient: ConversableAgent,
53
-
):
54
-
return recipient.last_message(sender)["content"]
55
-
```
56
-
"summary_prompt" can be used to specify the prompt used to extract a summary when summary_method is "reflection_with_llm".
57
-
Default is None and the following default prompt will be used when "summary_method" is set to "reflection_with_llm":
58
-
"Identify and extract the final solution to the originally asked question based on the conversation."
59
-
"carryover" can be used to specify the carryover information to be passed to this chat.
60
-
If provided, we will combine this carryover with the "message" content when generating the initial chat
61
-
message in `generate_init_message`.
42
+
- clear_history (bool): whether to clear the chat history with the agent. Default is True.
43
+
- silent (bool or None): (Experimental) whether to print the messages for this conversation. Default is False.
44
+
- cache (Cache or None): the cache client to be used for this conversation. Default is None.
45
+
- max_turns (int or None): the maximum number of turns for the chat. If None, the chat will continue until a termination condition is met. Default is None.
46
+
- "message" needs to be provided if the `generate_init_message` method is not overridden.
47
+
Otherwise, input() will be called to get the initial message.
48
+
- "summary_method": a string or callable specifying the method to get a summary from the chat. Default is DEFAULT_summary_method, i.e., "last_msg".
49
+
- Supported string are "last_msg" and "reflection_with_llm":
50
+
when set "last_msg", it returns the last message of the dialog as the summary.
51
+
when set "reflection_with_llm", it returns a summary extracted using an llm client.
52
+
`llm_config` must be set in either the recipient or sender.
53
+
"reflection_with_llm" requires the llm_config to be set in either the sender or the recipient.
54
+
- A callable summary_method should take the recipient and sender agent in a chat as input and return a string of summary. E.g,
55
+
```python
56
+
def my_summary_method(
57
+
sender: ConversableAgent,
58
+
recipient: ConversableAgent,
59
+
):
60
+
return recipient.last_message(sender)["content"]
61
+
```
62
+
"summary_prompt" can be used to specify the prompt used to extract a summary when summary_method is "reflection_with_llm".
63
+
Default is None and the following default prompt will be used when "summary_method" is set to "reflection_with_llm":
64
+
"Identify and extract the final solution to the originally asked question based on the conversation."
65
+
"carryover" can be used to specify the carryover information to be passed to this chat.
66
+
If provided, we will combine this carryover with the "message" content when generating the initial chat
0 commit comments