-
Notifications
You must be signed in to change notification settings - Fork 5.3k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Change last_n_messages to default to 'auto' #1356
Conversation
Codecov ReportAttention:
Additional details and impacted files@@ Coverage Diff @@
## main #1356 +/- ##
===========================================
- Coverage 32.45% 14.18% -18.28%
===========================================
Files 41 39 -2
Lines 4905 4899 -6
Branches 1119 1120 +1
===========================================
- Hits 1592 695 -897
- Misses 3187 4178 +991
+ Partials 126 26 -100
Flags with carried forward coverage won't be shown. Click here to find out more. ☔ View full report in Codecov by Sentry. |
Just to add some data to support this move. I used AutoGenBench to run a random 10% subsample of HumanEval through the GroupChatThreeAgents_Distractor scenario in two conditions: last_n_messages = 1 (the current default), and last_n_messages = "auto". I used GPT-3.5-turbo-16k. The results are as follows:
|
…/autogen into change_last_n_messages_default
* Change last_n_messages to default to 'auto' * Added last_n_messages validation, as per Eric's comment. --------- Co-authored-by: Davor Runje <[email protected]>
* Change last_n_messages to default to 'auto' * Added last_n_messages validation, as per Eric's comment. --------- Co-authored-by: Davor Runje <[email protected]>
* Change last_n_messages to default to 'auto' * Added last_n_messages validation, as per Eric's comment. --------- Co-authored-by: Davor Runje <[email protected]>
Why are these changes needed?
An 'auto' mode was added to the last_n_messages code execution configuration in #693. Since then it has been my top recommended setting for people struggling with GroupChat. Basically, if last_n_messages is set too low (e.g., the current default of 1), then code execution can break if another agent speaks between when code is written and when it is set to be executed. 'Auto' avoid this problem.
In non-GroupChat settings, last_n_message = 'auto' and last_n_message = 1 work identically. There would be no change in behavior.
TL;DR;
Though 'auto' can vastly improve GroupChat performance (e.g., see my experiments with HumanEval), it is still not the default. This PR changes that.
Related issue number
#693
Checks