Inconcistency between Responses API and Chat Completions API on rate limit errors

[aloumakos]

Confirm this is an issue with the Python library and not an underlying OpenAI API

• This is an issue with the Python library

Describe the bug

I noticed that rate limit errors are raised differently in the two APIs during streaming. In chat completions the error is raised and retried by the client itself on the initial call, before reading the stream. On the other hand, the first call to responses.create(...) returns a stream object successfully and then upon reading the stream, an APIError is raised indicating a rate limit. As a result, the whole retrying logic in the client is completely skipped and the user has to implement their own. It is also unclear whether an error could be raised mid-stream which adds to the complexity of the fix.

After inspecting the client's code, I initially thought that this is not a library issue but a misalignment with the model's service. I contacted Microsoft support since they host our models but they insisted I raise an issue here.

To Reproduce

To reproduce this you would need an Azure hosted OpenAI model ( I have personally already tried this with gpt-4o and gpt-5 ). Lower your token rate limit threshold to a low enough value and use the below snippet.

Code snippets

from openai import OpenAI

client = OpenAI(  
  base_url = "your-endpoint-v1",
  api_key="your-api-key"
)

response = client.responses.create(
    model="your-model",
    store=False,
    stream=True,
    input='a long enough prompt'*10000 # to hit a rate limit
)

for event in response:
   continue # an APIError is raised here

OS

Linux, Windows

Python version

Python >=3.11.9

Library version

openai v1.109.1

[karpetrosyan]

Yeah, it seems like we only retry for the initial requests, and not when the response is rate-limiting in the middle of streaming. Could you provide more information about what you receive in the APIError - message, status, etc.?

[aloumakos]

@karpetrosyan This is the local state after I set a breakpoint at line 193 of _streaming.py.

[karpetrosyan]

I don't think it would be simpler for users if we added automatic retries mid-stream due to rate limits. I think this should be the expected behavior.

[aloumakos]

Does Chat Completions API retry mid-stream? I have never had a rate limit error show up mid-stream using Chat Completions, the error always comes before I even start reading the stream, exactly when I create the stream object and there the client retries it automatically. Meanwhile, for Responses API the stream object is always successfully created and I get a rate limit error right after I start reading the stream ( it has never happened mid-stream yet, always at the very start of the stream). Is this inconsistency in behavior between the two APIs expected? The client never retries a rate limit error when using the Responses API.

[karpetrosyan]

They both stream via the same API, so the behavior should be the same. Neither can retry mid-stream because the user has already consumed part of it, and it's unclear what retry would even mean in this context. I don't think there's an inconsistency at the SDK level, though there might be an issue with the OpenAI part, which isn't tracked in this repo.