-
Notifications
You must be signed in to change notification settings - Fork 5.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Support for WebSockets, streaming responses to a frontend [Feature Request]: #1199
Comments
Hi @tyler-suard-parker ,
|
Just a suggestion: by adding "TERMINATE" at the beginning of the last message, rather than at the end of the last message, we could know which message is the final one, so that message could be streamed to the frontend. |
@Risingabhi that is an interesting approach, I agree, it might be a good idea to modify the print statement to send messages on the websockets instead. Thank you! |
can anyone share code for how to implement streaming for external application. |
@Risingabhi @tyler-suard-parker @super-syan we are refactoring and hoping to make it very easy to add streaming. See #1240 and please give your feedback and comments. |
@ekzhu I am not familiar with hooks, I was actually hoping for something more like SSE. |
What is SSE? As I mentioned in the other discussion #1290 , the goal of the refactoring effort is to modularize functionalities as middleware, so it becomes very easy to extend an agent with things like emitting a message to a frontend receiver. |
@tyler-suard-parker the hooks got replaced by Middleware pattern and this seems like the best design pattern to use. For SSE, you basically need a generator that can be easily implemented using the Middleware pattern. We implement it as well while implementing other features. |
@ekzhu Sorry about that, SSE is server-sent events. |
@Risingabhi Could you give some more details about your implementation? Where are you declaring the server, client, etc? |
I just tried @Risingabhi 's implementation, it does not work with streaming, it just sends the complete message to a socket, not the chunks. |
I found a solution for this. I am using a singleton class in a file in the autogen/oai directory. I import that singleton class into my top script, and I store my websocket as a variable in that class. I also modify autogen/oai/client.py to import that same script and that same websocket, which I can then use to stream data from the _completions_create function in that file. |
I think we need to implement this in the framework for everyone else. |
@davorrunje I agree and I would be happy to do it, but my use case is very specific. I am running on Azure Web Apps and returning a stream to an Azure Teams App. Also I am not quite sure how to disable the streaming to a UI if it is not needed. |
Let's keep it open and I think I might take it soon. Any help would be appreciated though :) |
Hi I have implemented streaming for my app checkout my repository you can disable and enable using use_socket Boolean. |
I am currently trying to work with react frontend. I already tried it with
panel. It worked fine. Will let you know soon.
…On Thu, 18 Jan, 2024, 6:31 am tyler-suard-parker, ***@***.***> wrote:
@Risingabhi <https://github.com/Risingabhi> Could you give some more
details about your implementation? Where are you declaring the server,
client, etc?
—
Reply to this email directly, view it on GitHub
<#1199 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AEH5EVWAVK6QGN7QX6HS5ULYPBX65AVCNFSM6AAAAABBVFUTIKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTQOJXGU4DANZTGI>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
@Risingabhi |
All, I finally bit the bullet and figured out how to do this, I just instantiated a websocket at the top level, and then passed that down as a parameter through every function between the top level and the _create_completion function in oai/client.py. It works! |
Awesome, thanks for the update!
…On Fri, 26 Jan, 2024, 12:37 am tyler-suard-parker, ***@***.***> wrote:
All, I finally bit the bullet and figured out how to do this, I just
instantiated a websocket at the top level, and then passed that down as a
parameter through every function between the top level and the
_create_completion function in oai/client.py. It works!
—
Reply to this email directly, view it on GitHub
<#1199 (comment)>,
or unsubscribe
<https://github.com/notifications/unsubscribe-auth/AEH5EVREIEFRK56MGLOM6D3YQKUQLAVCNFSM6AAAAABBVFUTIKVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMYTSMJQHAYTONJXHA>
.
You are receiving this because you were mentioned.Message ID:
***@***.***>
|
Is your feature request related to a problem? Please describe.
I am using a Microsoft Teams App for a frontend, and hosting Autogen on my backend. When I send a message to Autogen from my frontend, Autogen does some processing and then sends back the final answer. This can take up to 2 minutes, which is too long for my customers. The vast majority of that time is taken up by GPT-4 generating an answer. I would like it if the final answer generated by Autogen could be streamed back to the frontend, using websockets or another protocol that is compatible with Microsoft Teams Apps. This would give my users something to look at immediately, rather than waiting a long time for the complete answer to pop up.
Describe the solution you'd like
An easy, plug-and-play solution for Microsoft Teams Apps, that allows Autogen to stream information to the Teams App.
Additional context
No response
The text was updated successfully, but these errors were encountered: