Streaming is broken

[randomsnowflake]

Code

import Foundation
import OpenAI

let openAI = OpenAI(apiToken: "sk-...")

let query = OpenAI.ChatQuery(model: .gpt3_5Turbo, messages: [.init(role: "user", content: "hi!")], stream: true)
let result = try await openAI.chats(query: query)

Exception

Swift/ErrorType.swift:200: Fatal error: Error raised at top level: Swift.DecodingError.dataCorrupted(Swift.DecodingError.Context(codingPath: [], debugDescription: "The given data was not valid JSON.", underlyingError: Optional(Error Domain=NSCocoaErrorDomain Code=3840 "Invalid value around line 1, column 0." UserInfo={NSDebugDescription=Invalid value around line 1, column 0., NSJSONSerializationErrorIndex=0})))

2023-03-08 14:37:01.246548+0100 CLIChat[81308:5267103] Swift/ErrorType.swift:200: Fatal error: Error raised at top level: Swift.DecodingError.dataCorrupted(Swift.DecodingError.Context(codingPath: [], debugDescription: "The given data was not valid JSON.", underlyingError: Optional(Error Domain=NSCocoaErrorDomain Code=3840 "Invalid value around line 1, column 0." UserInfo={NSDebugDescription=Invalid value around line 1, column 0., NSJSONSerializationErrorIndex=0})))

It works with stream: false

[Krivoblotsky]

Hey, @randomsnowflake.
Thanks for bringing this to our attention.
Looks like streaming is unsupported for now and we need to implement server-sent events for this.

[longseespace]

I hope this will be implemented soon. Meanwhile, you can use this alternative (only for streaming - I'm using both clients in my app)

https://github.com/nate-parrott/openai-streaming-completions-swift

[randomsnowflake]

I already use it. I think it's also the only Swift OpenAI API with streaming support.

However, have a look at my issue report here: Missing Model Parameters and No Error Message ... it's not as mature as I would like. Looking forward to MacPaw's implementation of streaming.

[Krivoblotsky]

I know that @rshimoda is on his way to adding streaming support here, so I believe it must be implemented shorty 👌

[rshimoda]

Hey there, I'll do my best to implement this soon 😉

[DJBen]

@rshimoda Hi Sergi do you have a timeline for the implementation? Let me know if you need any help or have any WIP branch.

[Krivoblotsky]

@rshimoda 🙏

[DJBen]

I am exploring a solution right now - it looks like a bigger work than I expect, and it is more akin to a separate project than 'fixing' the streaming.

We might need to rely on other EventSource libraries like https://github.com/launchdarkly/swift-eventsource. Let me know if you think is okay to add dependency to another library.

[Krivoblotsky]

Hey, @DJBen and everyone.
I have some implementation in mind that doesn't require any 3rd party libraries. I have implemented PoC that supports streaming for Chats API with minimal changes required.
Could you take a look at #57?

It has several weak points, such as optional delta and message in ChatResult, but overal looks good to me.
Public interfaces is untouched, but the completion coulbe called multiple times with the delta:

openAI.chats(query: .init(model: .gpt3_5Turbo, messages: [.init(role: .user, content: "Who is Taras Shevchenko?")], stream: true)) { result in
    switch result {
    case .success(let res):
        //called multiple times
        if let messageDelta = res.choices.first?.delta?.content {
            print(messageDelta)
        }
    case .failure(let error):
        print(error)
    }
}

Any feedback is very appreciated 🙏

[Krivoblotsky]

Almost there: #57 🤞

[Krivoblotsky]

Fixed by #57