v4.0.0 Beta

[schnerd]

Important

OpenAI v4 has launched and is no longer in beta! Give it a spin with npm install openai and check out the migration guide

---

We're working on a new major version of our SDK and are excited to have you test it!

Please use this discussion page as a place to report issues and feedback about the new release.

What's new

• Streaming support for completions and chat completions!
• Automatic retries (defaults to 2)
• Improved typescript types for responses & errors
• Support for ESM, cloudflare workers, vercel edge functions, deno
• Improved latency through reusing TCP connections
• Simpler initialization logic

View the v4 API

Things we'd love feedback on

• Does streaming work reliably?
• Do retries work reliably?
• Are there any issues with the typescript types?
• Are there any regressions compared to v3?
• Any other things you love or that bother you?

Getting started

npm install [email protected]

Migration guide

➡️ View v3 to v4 migration guide

[iemesowum]

This is great news! Will you be adding token counting for streaming support, perhaps on the final SSE chunk? This is needed for accurate usage-based billing.

I've adapted the following code from the web which produces similar results to your billing reports for gpt-3.5-turbo. It's not accurate for gpt-4. It uses https://www.npmjs.com/package/gpt-3-encoder.

const calculateTokens = (messages: Array<ChatCompletionRequestMessage>) => {
  let count = 0;
  messages.slice(0, -1).forEach((message) => {
    count += 4;
    let encodedContent = encode(message.content);
    let encodedRole = encode(message.role);
    count += encodedContent.length;
    count += encodedRole.length;
  });
  let last = messages.at(messages.length - 1) as ChatCompletionRequestMessage;
  let encodedContent = encode(last.content);
  count += encodedContent.length;
  return count;
};

[schnerd]

@iemesowum We can look into this separately – however this would be an API-level improvement (which would automatically apply to all SDKs, including this one)

[RonanKMcGovern]

yes, this is very much needed. Without accurate usage, it's harder to set pricing and determine a business model for building apps that use openai.

[viviando1110]

Would be great to have usage report for streaming, this is very much needed

[gregreindel]

This is great, appreciate the simplified config and error handling.

I had some type issue in completions(chat) that was triggered by a change in how you are handing streaming. I dug in a little, and feel like the typing in there a bit redundant at times, and maybe slightly incomplete.

Ultimately it feels like CompletionCreateParams should just be a base interface and have the stream-enabled variant extend it. I don't need to maintain this, but it is a lot of duplicate code, with the only subtle differences I see being the stream param and some small bits of the output/event respectively.

As an example, one minor difference between ChatCompletion and ChatCompletionEvent is the usage property. In ChatCompletion, usage is optional (but I'm pretty sure i'll always be there in this case) while in ChatCompletionEvent usage doesn't exist. Similarly, in ChatCompletion*.Choices, the difference there is delta, which is optional in ChatCompletionEvent (again seems like it's there though in this context), and non-existent in ChatCompletion. These could simply extend a base that had an optional param. The optional types make me think this was the intention at some time but then went a more complex route? The namespaces in there also seems overkill, esp as its unused outside of that file.

[Akaryatrh]

@schnerd I'm running into the following error with the v4.0.0-beta.0

This expression is not constructable.
Type 'typeof import("stackblitz:/node_modules/openai/dist/cjs/index")' has no construct signatures.

Issue is reproduced here: https://stackblitz.com/edit/node-ddzjcn?file=src%2Findex.ts

[rattrayalex]

Thank you for the repro! I'm not sure we'll be able to support stackblitz out of the gate with this version but we'll take a look soon.

[Akaryatrh]

It's also happening on my local machine, the issue is not scoped to Stackblitz 😅

[rattrayalex]

Hey @Akaryatrh, thanks for reporting this – we've added proper ESM support in v4.0.0-beta.1! Can you give it a try and let us know how it works for you?

[Akaryatrh]

Hey @rattrayalex , sorry I missed your comment! Looks like issue is resolved since beta.1
Thanks a lot and nice job to the team!

[murrayju]

I don't believe that the CreateChatCompletionRequestStreaming type is exported on the namespace in a way that I can consume it. The chat completions seem to define the same interface as the regular completions, where only the latter is exported from the root namespace. I'm guessing that the root namespace is just missing a line for

export import Chat = API.Chat;

I'm expecting to be able to do something along the lines of

import type OpenAI from 'openai';

export interface AiChatMessage
  extends OpenAI.Chat.CompletionCreateParams.CreateChatCompletionRequestStreaming {
  id: string;
}

[rattrayalex]

Thanks for reporting, we'll get that fixed shortly!

[rattrayalex]

This is fixed in v4.0.0-beta.1!

[lucjac]

Thanks for your great work!

I'm getting this error when running in streaming mode on a NextJS vercel edge function:

Error: The edge runtime does not support Node.js 'stream' module.

Do you know of any way to make this package work with the edge function limitations?

[RobertCraigie]

Thanks for the report! We'll look into it.

[RobertCraigie]

@lucjac it looks like your implementation was just missing encoding the completion content into Uint8Arrays, e.g.

+   const encoder = new TextEncoder();
    const completion = await openAI.chat.completions.create({
      model,
      temperature,
      messages,
      stream: true,
    })
    const stream = new ReadableStream({
      async start(controller) {
        for await (const part of completion) {
          console.log(part.choices[0].delta.content)
-         controller.enqueue(part.choices[0].delta.content)
+         controller.enqueue(encoder.encode(part.choices[0].delta.content))
        }
      },
    })
    return new Response(stream)

We're looking into ways we can provide this experience out of the box and will hopefully have an update soon!

[lucjac]

ahh that works, thanks a lot @RobertCraigie !

[lili21]

I keep getting the build error

`export const config` in /vercel/path0/app/api/chat/route.ts is deprecated. Please change `runtime` property to segment export config. See https://nextjs.org/docs/app/api-reference/file-conventions/route-segment-config

[rattrayalex]

@lili21 that doesn't seem to be related to this library, but if it is, can you please open a separate issue with more details of your code and your configuration?

[miguelduarte42]

I'm having some issues using fileFromPath to load the audio. Here's my code and error:

async speechToText(fileData, prompt, model = 'whisper-1') {
    let filename = tmp.tmpNameSync();
    try {
      await fsp.writeFile(filename, fileData.buffer);
      let file = await OpenAI.fileFromPath(filename);
      const response = await this.openai.audio.transcriptions.create({
        model,
        file,
        prompt,
      });
      return response;
    } finally {
      fsp.unlink(filename).catch(err => {
        console.log(err);
      })
    }
  }

TypeError: null is not a valid form data value, if you want to pass null then you need to use the string 'null'
    at addFormValue (/Users/miguelduarte/Desktop/homemd-server/node_modules/openai/dist/cjs/core.js:766:11)
    at /Users/miguelduarte/Desktop/homemd-server/node_modules/openai/dist/cjs/core.js:752:56
    at Array.forEach (<anonymous>)
    at createForm (/Users/miguelduarte/Desktop/homemd-server/node_modules/openai/dist/cjs/core.js:752:30)
    at multipartFormRequestOptions (/Users/miguelduarte/Desktop/homemd-server/node_modules/openai/dist/cjs/core.js:747:37)
    at Transcriptions.create (/Users/miguelduarte/Desktop/homemd-server/node_modules/openai/dist/cjs/resources/audio/transcriptions.js:14:46)

I think the result from fileFromPath is coming as null.

[miguelduarte42]

I thought the prompt was optional. Is that not the case?

[rattrayalex]

If you don't want to pass prompt, you need to pass undefined instead of null. Does that work?

Sorry the error message here isn't great.

[rattrayalex]

Note that we're also currently working on better alternatives to fileFromPath and hope to have something we'll seek your feedback on shortly!

[RobertCraigie]

No you are correct, the prompt is optional. However, there is a bug here where we don't strip out undefined properties that you pass to multipart form data. Will get a fix out soon!

[rattrayalex]

Fix is out – please upgrade to v4.0.0-beta.1!

[ykdojo]

I tried streaming and it's working well so far, thank you. Also thank you to @RobertCraigie for letting me know about it.

Would it be helpful to document an example directory for this somewhere?

[ykdojo]

Got it, thank you!

[hiroshinishio]

It worked!
Incidentally, can I set up a callback to be triggered by being aborted?

[hiroshinishio]

APIUserAbortError in node_modules\openai\src\error.ts can be used as a callback?

[rattrayalex]

You can listen to the 'abort' event on the AbortSignal like so:

stream.controller.signal.addEventListener('abort', () => {…});

[hiroshinishio]

That is better!

[RonanKMcGovern]

I'm hitting an error:

for await (const part of stream) {
                             ^

TypeError: stream is not async iterable

I'm using commonJS for import:

require('dotenv').config()

const OpenAI = require('openai');

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY
});

and here's the relevant code:

const stream = openai.chat.completions.create({
        model: 'gpt-3.5-turbo-16k-0613',
        messages: messages,
        functions: functions,
        function_call: "auto",
        stream: true,
    });
  
    let content = '';
  
    for await (const part of stream) {
        const responseMessage = part.choices[0].delta;
  
        if (responseMessage.function_call) {
            const functionName = responseMessage.function_call.name;
            const functionArgs = JSON.parse(responseMessage.function_call.arguments);
            const functionResponse = await availableFunctions[functionName](
                ...Object.values(functionArgs)
            );
            messages.push(responseMessage);
            messages.push({
                "role": "function",
                "name": functionName,
                "content": functionResponse,
            });
  
            const secondStream = openai.chat.completions.create({
                model: 'gpt-3.5-turbo-16k-0613',
                messages: messages,
                stream: true,
            });
  
            for await (const part2 of secondStream) {
                content += part2.choices[0].delta;
            }
        } else {
            content += responseMessage;
        }
    }
  
    return content;
  }

[musqdp]

const OpenAI = require('openai');

// Set your API key
const apiKey = 'API_KEY';

// Create an instance of OpenAI using your key
const openaiInstance = new OpenAI({
  apiKey: apiKey,
});

// Function for processing data from the stream
async function processStream(stream) {
  for await (const chunk of stream) {
    if (chunk.object === 'chat.completion.chunk') {
      // Display data from the chunk
      console.log('Received chunk:', chunk.choices[0].delta.content);
    }
  }
}

// Call the chat completion with a data stream
async function generateChat() {
  const stream = await openaiInstance.chat.completions.create({
    model: 'gpt-3.5-turbo-16k-0613',
    messages: [{ role: 'user', content: 'Hello world' }],
    stream: true,
  });

  // Process and display data from the stream
  await processStream(stream);
}

generateChat();

[rattrayalex]

@musqdp what was the purpose of pasting this code – is there an issue with it, or were you providing it as an example of how to use stream: true?

[musqdp]

This just example how to use stream. I didn't find any examples in the README.md of this library.

[RobertCraigie]

@musqdp here's our README example: https://github.com/openai/openai-node/tree/v4#streaming-responses, you may have missed it because it's currently not in the default branch.

[PiotrWarzachowski]

I wanted to contribute a bit by making this open source project, where chat response is beeing streamed in real time you can see it by going here : https://github.com/PiotrWarzachowski/DocMind Feel free to check it out, its public on vercel. You can leave a star ⭐ if you want 💓. Have a good year

[MasterOdin]

Similar to #182 (comment), I think that CreateCompletionRequestNonStreaming type should also be exposed. I'm trying to use OpenAI.CompletionCreateParams, but I'm getting an error from TS that I'm not sure how to resolve.

My original code was:

async function getCompletion(
  args: CreateChatCompletionRequest,
): Promise<CompletionResult> {
  try {
    const completion = (await openai.createChatCompletion(args)).data;
    return { completion };
  } catch (error) {
    return { error: new Error((error as Error).message) };
  }
}

which I've replaced with:

async function getCompletion(
  args: OpenAI.CompletionCreateParams,
): Promise<CompletionResult> {
  try {
    const completion = await openai.chat.completions.create(args);
    return { completion };
  } catch (error) {
    return { error: new Error((error as Error).message) };
  }
}

and I get a tsc error for my usage of args:

No overload matches this call.
  Overload 1 of 2, '(body: CreateChatCompletionRequestNonStreaming, options?: RequestOptions<Readable | Record<string, unknown>> | undefined): Promise<...>', gave the following error.
    Argument of type 'CompletionCreateParams' is not assignable to parameter of type 'CreateChatCompletionRequestNonStreaming'.
      Property 'messages' is missing in type 'CreateCompletionRequestNonStreaming' but required in type 'CreateChatCompletionRequestNonStreaming'.
  Overload 2 of 2, '(body: CreateChatCompletionRequestStreaming, options?: RequestOptions<Readable | Record<string, unknown>> | undefined): Promise<...>', gave the following error.
    Argument of type 'CompletionCreateParams' is not assignable to parameter of type 'CreateChatCompletionRequestStreaming'.
      Property 'messages' is missing in type 'CreateCompletionRequestNonStreaming' but required in type 'CreateChatCompletionRequestStreaming'.

with no clear way to resolve it.

[RobertCraigie]

Hey @MasterOdin, sorry about this! We should be getting an update out later today / tomorrow that exposes the params types correctly among many other improvements.

[rattrayalex]

This is now available in v4.0.0-beta.1!

[RobertCraigie]

Note that currently you have to access the type like this:

export async function getCompletion(
  args: OpenAI.Chat.CompletionCreateParams.CreateChatCompletionRequestNonStreaming,
) {
  // ...
}

But soon we will update the SDK so that you can just reference OpenAI.Chat.CompletionCreateParamsNonStreaming instead.

[TMarafon]

Amazing. This new version tackles several issues we were having to code ourselves. Thanks so much.

Regarding your last question: "Any other things you love or that bother you?", I'd love to have some utilities to help us deal with input sanitization. Safety is a main concern and any features that can make our applications safer are very welcome.

[dosstx]

Hopefully not too off-topic, but does anyone have thoughts on Vercel's AI open AI SDK: https://sdk.vercel.ai/docs/guides/openai vs the one being discussed here? pros and cons and benefits of using this Node version? The vercel version is optimized for streaming "on the edge" (non-Node, I think).

[RobertCraigie]

Yup it works!

Followed the quickstart you linked and modified it to use the v4 SDK:

import OpenAI from 'openai';
import { OpenAIStream, StreamingTextResponse } from 'ai';
import { NextRequest } from 'next/server';
 
export const config = {
  runtime: 'edge',
  unstable_allowDynamic: [
    // This is currently required because `qs` uses `side-channel` which depends on this.
    '/node_modules/function-bind/**',
  ],
};
 
export default async (request: NextRequest) => {
  const openai = new OpenAI();
 
  // Extract the `messages` from the body of the request
  const { messages } = await request.json();
 
  // Ask OpenAI for a streaming chat completion given the prompt
  const streamResponse = await openai.chat.completions.create({
    model: 'gpt-3.5-turbo',
    stream: true,
    messages,
  });

  // @ts-expect-error openai currently use node-fetch types which are incompatible with web types
  const stream = OpenAIStream(streamResponse.response);

  // Respond with the stream
  return new StreamingTextResponse(stream);
}

Note the ts-expect-error is an unfortunate byproduct of us still using types from node-fetch instead of web types - will fix soon!

[dosstx]

When trying to use Nuxt and the above code, I get the following server error:

{
    "url": "/api/chat",
    "statusCode": 500,
    "statusMessage": "",
    "message": "responseBodyStream.pipeThrough is not a function"
}

My code:

import OpenAI from 'openai'
import { OpenAIStream, StreamingTextResponse } from 'ai'

const config = useRuntimeConfig()
const openai = new OpenAI({
  apiKey: config.openaiApiKey
})

export default defineEventHandler(async (event: any) => {

  // Extract the `prompt` from the body of the request
  const { messages } = await readBody(event)

  // Ask OpenAI for a streaming chat completion given the prompt
  const streamResponse = await openai.chat.completions.create({
    model: 'gpt-3.5-turbo',
    stream: true,
    messages: messages.map((message: any) => ({
      content: message.content,
      role: message.role
    }))
  })

  // Convert the response into a friendly text-stream
  const stream = OpenAIStream(streamResponse.response)

  // Respond with the stream
  return new StreamingTextResponse(stream)
})

If any Nuxt experts have any ideas, let me know. Thanks.

[RobertCraigie]

@dosstx what runtime are you using? I suspect this may be an issue with the Vercel SDK and older runtimes, https://developer.mozilla.org/en-US/docs/Web/API/ReadableStream/pipeThrough#browser_compatibility

[alii]

This happens when using a non "edge-runtime" route on Next.js (e.g. the older lambda-style Node.js api routes in pages dir). I think this is an issue with the Vercel AI SDK, but frustrating as I am running into the issue now.

[rattrayalex]

Ah, sorry to hear that. It does sound like an issue to report to Vercel (whether on NextJS or their AI SDK, I'm not sure).

[dosstx]

@RobertCraigie I think that's part of the problem. I am confused about the runtime used for Nuxt's new server engine (Nitro).

Nitro produces a standalone server dist that is independent of node_modules. "The output contains runtime code to run your Nuxt server in any environment" : https://github.com/unjs/nitro

[RobertCraigie]

@dosstx ah it looks like people have ran into issues using Nuxt with the AI SDK before as well: vercel/ai#196 and that Nitro doesn't support streaming responses yet anyway: nitrojs/nitro#1327

[Aietes]

I think you can stream in Nuxt using return sendStream(event, stream) in defineEventHandler, or just write to event.node.res, while using $fetch with responseType: 'stream'.

[dosstx]

@Aietes : Did you ever get Nuxt 3.6.5 to work with OpenAI Node SDK version 4? Can you share how you are streaming data? Thank you!

[davidjbng]

I was unable to configure v4 (4.0.0-beta.2) to work with Azure OpenAI Service.

// Azure OpenAI in v3
const openai = new OpenAIApi(new Configuration({
    apiKey,
    basePath: `https://${deploymentName}.openai.azure.com/openai/deployments/${model}`,
    baseOptions: {
       headers: {
         "api-key": apiKey,
       },
       params: {
         "api-version": apiVersion,
       },
     },
  }))

// Not working: Azure OpenAI in v4
const openai = new OpenAI({
  apiKey,
  baseURL: `https://${deploymentName}.openai.azure.com/openai/deployments/${model}`,
  defaultHeaders: {
    "api-key": apiKey,
  },
  // where to specify default param `api-version`?
});

I was actually receiving a 404 NotFoundError when calling openai.chat.completions.create(...) with v4 so I am unsure if baseURL is even working.

Happy to test out new versions to confirm they can be configured to talk to the Azure API.

[rattrayalex]

Thanks for the report! We'll add defaultQueryParams or equivalent soon.

[gdl-reif]

As a workaround, you can add options to the create call:

await api.chat.completions.create({
        //  ...
    }, { 
        query : { "api-version": "2023-06-01-preview" },
        headers: {'api-key': process.env.OPENAI_AZURE_API_KEY }
    });

[jmn]

I am seeing this error with beta.2 and NextJS:

[1] - error Error [TypeError]: Cannot read properties of undefined (reading 'FileObjectsPage')
[1]     at <unknown> (webpack-internal:///(sc_server)/./node_modules/.pnpm/[email protected]/node_modules/openai/resources/index.mjs:8)
[1]     at Module.FileObjectsPage (webpack-internal:///(sc_server)/./node_modules/.pnpm/[email protected]/node_modules/openai/resources/index.mjs:8:108)
[1]     at eval (webpack-internal:///(sc_server)/./node_modules/.pnpm/[email protected]/node_modules/openai/resources/files.mjs:59:69)
[1]     at eval (webpack-internal:///(sc_server)/./node_modules/.pnpm/[email protected]/node_modules/openai/resources/files.mjs:60:3)
[1]     at (sc_server)/./node_modules/.pnpm/[email protected]/node_modules/openai/resources/files.mjs (file:///home/jmn/git/project/.next/server/app/api/completion/route.js:178

If I edit openai/resources/index.mjs and remove the exports of FileObjectsPage, FineTunesPage and ModelsPage, the package works as expected.

[RobertCraigie]

@jmn we believe we have a fix but we haven't been able to reproduce the error you're seeing, could you please share your setup?

[jmn]

Unfortunately I can not share my private repo, but I am using a fork of this repo https://github.com/shadcn/taxonomy with mostly the same settings. When I installed beta.2 and tried to call the api route I got the above error. However I do not get the error with beta.1.

Let me know if there is something else you need.

[dan-kwiat]

Hey @rattrayalex @RobertCraigie great to see you working on this - I'm the author of openai-edge. I'm experiencing the same error above when using the Next edge runtime. Minimal reproduction here: https://github.com/dan-kwiat/next-openai-v4

[RobertCraigie]

Hey @dan-kwiat, thanks for the repo, I can reproduce the error! I can also confirm that our patch does fix the issue. We'll be publishing a new release soon that includes this fix.

Regarding the pnpm build error mentioned in the README, this is a redundant error from Next.js which has also been fixed! You can try it out by installing the canary version or until v13.4.9 is fully released you will need to add function-bind to your edge config, e.g.

export const config = {
  runtime: 'edge',
  unstable_allowDynamic: [
    '/node_modules/function-bind/**',
  ],
}

https://nextjs.org/docs/messages/edge-dynamic-code-evaluation

[dan-kwiat]

That's great @RobertCraigie thanks for the update. The build error was whilst including function-bind in the config, but I'll wait for the next releases before looking into it again.

[MRYingLEE]

I am waiting for the stream mode. Just be curious whether the streaming feature can be used in Jupyter Notebook with JavaScript kernel. If so, a demo is welcomed.

[rattrayalex]

Yes, the beta supports streaming; here's a demo script. Please give it a try and let us know if you run into any issues, with a full code example and traceback. We have not yet tested in a Jupyter Notebook.

[alvis]

Another issue with using v4 with next js. In next js, fetch is patched and it additionally allow passing {next: { revalidate: number }} to RequestInit. However, currently there is no way to pass this options with v4 nor any other headers (e.g. for tracing purpose).

Would it be possible to allow us to pass any of these extra in future releases?

[rattrayalex]

Thanks for reporting this @alvis , please let us know how the workaround @jedwards1211 mentioned works for you. We'll explore other methods of turning off or customizing Next's data cache.

[alvis]

Thanks @jedwards1211 & @rattrayalex. Using a custom fetch function works. In addition, if anyone want to control the cache TTL for individual page, here is a workaround I figured out.

const open = new OpenAI({
  fetch // use the global fetch supplied by next js rather than the default node-fetch
})

Then on each page

const fetchCache = 'force-cache'; // by default on fetch on static pages are cached, you may want to set it to 'force-cache' if you're on a dynamic page
const revalidate = 3600; // or 'force-cache' or any number

The only caveat is that revalidate specify the TTL of all fetch calls on the page. I can't figure out anyway to set TTL for individual call, but probably it's not a big deal.

See https://nextjs.org/docs/app/api-reference/file-conventions/route-segment-config

[rattrayalex]

Interesting, thank you for sharing. This is an interesting design by Vercel… Does it work to pass a Cache-Control: no-store header instead, like this?

const open = new OpenAI({
  fetch // use the global fetch supplied by next js rather than the default node-fetch
});
const completion1 = await openai.chat.completions.create(params, { headers: { 'Cache-Control': 'no-store' } });
const completion2 = await openai.chat.completions.create(params, { headers: { 'Cache-Control': 'no-store' } });
// check that 1 and 2 are different

(if so, it's more likely what we'd do, so your help in investigating this would be appreciated!)

[alvis]

@rattrayalex No it won't.

Based on the source code of the patched fetch, it doesn't take any Cache-Control header into account.

[rattrayalex]

Thanks for linking to the source there, I've opened a feature request with them: vercel/next.js#53504

[alvis]

Separately, I have another issue related to the async iterator streaming interface.

I've the following implementation for streaming response via the next js . It kinda work, except that most of the time the response is not streaming but returning the whole response at the very end. However, if I artificially add an async sleep, the response will stream as expected.

I guess it's related to the event loop???

const response = await openai.chat.completions.create({
    user: 'nextjs',
    model: 'gpt-3.5-turbo',
    stream: true,
    messages,
  });

  // const stream = OpenAIStream(response); // using OpenAIStream from 'ai' doesn't work either

  const encoder = new TextEncoder();
  const iterable = response[Symbol.asyncIterator]();
  const stream = new ReadableStream({
    async pull(controller) {
      const chunk = await iterable.next();

      if (chunk.done) {
        controller.close();
      } else {
        const value = chunk.value.choices[0].delta.content;

        if (value) {
          controller.enqueue(encoder.encode(value));
        }

        // await sleep(10); // if we don't sleep, streaming doesn't work
      }
    },
  });

  // Respond with the stream
  return new StreamingTextResponse(stream);

[rattrayalex]

Thank you for sharing! OOC, what happens if you remove the fetch, so it's just this:

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});
const fetchCache = 'force-cache';
…

[rattrayalex]

Do you want to use Vercel's data cache with your calls to this API, or not?

[alvis]

Yes & no. My application has two parts: analysis and chatbot. I want to cache any analysis result, but not chat message.
With the current approach, it's all or none.

[alvis]

Thank you for sharing! OOC, what happens if you remove the fetch, so it's just this:

const openai = new OpenAI({
  apiKey: process.env.OPENAI_API_KEY,
});
const fetchCache = 'force-cache';
…

This will make openai to use node-fetch rather than the global next-js patched fetch. i.e. no cache at all.

[rattrayalex]

Gotcha, thank you. Note that if you use Vercel's edge runtime, you'll get their fetch instead of node-fetch.

[OldManMeta]

I'm seeing an issue with a messages object:

Is there a conflicting type somewhere causing this, requiring a typed declaration?

i.e.

const messageCollection: CreateChatCompletionRequestMessage[]  = [
            { role: 'user', content: 'Whats the weather like in Boston?' },
          ];

[RobertCraigie]

@OldManMeta No unfortunately this is a quirk of TS. The error should go away if you define the role property like so:

{ role: 'user' as const }

[rattrayalex]

You can also do this, which will cover the stream: true part as well:

const params = {
  // …
} as const;

[Aietes]

I'm trying to use the Beta on Cloudflare pages, but I'm running into issues:

Cannot convert object to primitive value\n at OpenAI.buildRequest (functionsWorker-0.7372611095764459.js:8080:49) \n at OpenAI.request (functionsWorker-0.7372611095764459.js:8094:52) \n at OpenAI.post (functionsWorker-0.7372611095764459.js:8058:17) \n at hc.create (functionsWorker-0.7372611095764459.js:8305:17) \n at functionsWorker-0.7372611095764459.js:8448:75 \n at async Object.handler (functionsWorker-0.7372611095764459.js:2412:20) \n at async functionsWorker-0.7372611095764459.js:2455:7 \n at async Object.localFetch (functionsWorker-0.7372611095764459.js:3941:20)

Is there something specific I have to take care of when using the library on Cloudflare workers?

[johannschopplich]

@jedwards1211 Thank you very much for your efforts! I will try your patch while waiting for Nitro 2.6. Your help is much appreciated. 🙂

[rattrayalex]

Great! Just to be clear, you shouldn't need the patch now that we have a fix (which has now been released in 4.0.0, no longer beta!)

[jedwards1211]

@rattrayalex although this issue is fixed, I believe other issues are still likely without the patch due to the fact that Nitro 2.5.2 is unfortunately bundling node-fetch and other Node shims, and how they behave in the Cloudflare Worker environment is anyone's guess. Once Nitro 2.6.0 is released it should stop bundling any Node shims.

[dosstx]

@Aietes Do you still have this working? I'm struggling.

[rattrayalex]

@dosstx what error messages are you getting on openai@^4.0.0?

[OldManMeta]

Hey there - apologies if this is already covered by others.

I was just using the simple function calling example, and I really did not expect that the arguments value of the function to call would be streamed. Is that expected?

I guess if the response from the API is set to stream on your end, you're just stuffing the returned values into your model as well - be it content or arguments?

That's going to take a bit of strategy to make it work correctly.

The following is print screening chunks from the initial completions call:

Part=> {
  id: 'chatcmpl-7jkPGZUX96bV5olCIFt8Zv9iKUw1h',
  object: 'chat.completion.chunk',
  created: 1691137502,
  model: 'gpt-3.5-turbo-0613',
  choices: [ { index: 0, delta: [Object], finish_reason: null } ]
}
Delta=> {
  role: 'assistant',
  content: null,
  function_call: { name: 'get_current_weather', arguments: '' }
}
Content=> null
Inside Run Function
Calling Funtion Name=> get_current_weather
Funtion to Call=> [Function: get_current_weather]
Part=> {
  id: 'chatcmpl-7jkPGZUX96bV5olCIFt8Zv9iKUw1h',
  object: 'chat.completion.chunk',
  created: 1691137502,
  model: 'gpt-3.5-turbo-0613',
  choices: [ { index: 0, delta: [Object], finish_reason: null } ]
}
Delta=> { function_call: { arguments: '{\n' } }
Content=> undefined
Inside Run Function
Calling Funtion Name=>
Funtion to Call=> undefined
Part=> {
  id: 'chatcmpl-7jkPGZUX96bV5olCIFt8Zv9iKUw1h',
  object: 'chat.completion.chunk',
  created: 1691137502,
  model: 'gpt-3.5-turbo-0613',
  choices: [ { index: 0, delta: [Object], finish_reason: null } ]
}
Delta=> { function_call: { arguments: ' ' } }
Content=> undefined
Inside Run Function
Calling Funtion Name=>
Funtion to Call=> undefined
Part=> {
  id: 'chatcmpl-7jkPGZUX96bV5olCIFt8Zv9iKUw1h',
  object: 'chat.completion.chunk',
  created: 1691137502,
  model: 'gpt-3.5-turbo-0613',
  choices: [ { index: 0, delta: [Object], finish_reason: null } ]
}
Delta=> { function_call: { arguments: ' "' } }
Content=> undefined
Inside Run Function
Calling Funtion Name=>
Funtion to Call=> undefined
Part=> {
  id: 'chatcmpl-7jkPGZUX96bV5olCIFt8Zv9iKUw1h',
  object: 'chat.completion.chunk',
  created: 1691137502,
  model: 'gpt-3.5-turbo-0613',
  choices: [ { index: 0, delta: [Object], finish_reason: null } ]
}
Delta=> { function_call: { arguments: 'location' } }
Content=> undefined
Inside Run Function
Calling Funtion Name=>
Funtion to Call=> undefined
Part=> {
  id: 'chatcmpl-7jkPGZUX96bV5olCIFt8Zv9iKUw1h',
  object: 'chat.completion.chunk',
  created: 1691137502,
  model: 'gpt-3.5-turbo-0613',
  choices: [ { index: 0, delta: [Object], finish_reason: null } ]
}
Delta=> { function_call: { arguments: '":' } }
Content=> undefined
Inside Run Function
Calling Funtion Name=>
Funtion to Call=> undefined
Part=> {
  id: 'chatcmpl-7jkPGZUX96bV5olCIFt8Zv9iKUw1h',
  object: 'chat.completion.chunk',
  created: 1691137502,
  model: 'gpt-3.5-turbo-0613',
  choices: [ { index: 0, delta: [Object], finish_reason: null } ]
}
Delta=> { function_call: { arguments: ' "' } }
Content=> undefined
Inside Run Function
Calling Funtion Name=>
Funtion to Call=> undefined
Part=> {
  id: 'chatcmpl-7jkPGZUX96bV5olCIFt8Zv9iKUw1h',
  object: 'chat.completion.chunk',
  created: 1691137502,
  model: 'gpt-3.5-turbo-0613',
  choices: [ { index: 0, delta: [Object], finish_reason: null } ]
}
Delta=> { function_call: { arguments: 'Boston' } }
Content=> undefined
Inside Run Function
Calling Funtion Name=>
Funtion to Call=> undefined
Part=> {
  id: 'chatcmpl-7jkPGZUX96bV5olCIFt8Zv9iKUw1h',
  object: 'chat.completion.chunk',
  created: 1691137502,
  model: 'gpt-3.5-turbo-0613',
  choices: [ { index: 0, delta: [Object], finish_reason: null } ]
}
Delta=> { function_call: { arguments: ',' } }
Content=> undefined
Inside Run Function
Calling Funtion Name=>
Funtion to Call=> undefined
Part=> {
  id: 'chatcmpl-7jkPGZUX96bV5olCIFt8Zv9iKUw1h',
  object: 'chat.completion.chunk',
  created: 1691137502,
  model: 'gpt-3.5-turbo-0613',
  choices: [ { index: 0, delta: [Object], finish_reason: null } ]
}
Delta=> { function_call: { arguments: ' MA' } }
Content=> undefined
Inside Run Function
Calling Funtion Name=>
Funtion to Call=> undefined
Part=> {
  id: 'chatcmpl-7jkPGZUX96bV5olCIFt8Zv9iKUw1h',
  object: 'chat.completion.chunk',
  created: 1691137502,
  model: 'gpt-3.5-turbo-0613',
  choices: [ { index: 0, delta: [Object], finish_reason: null } ]
}
Delta=> { function_call: { arguments: '"\n' } }
Content=> undefined
Inside Run Function
Calling Funtion Name=>
Funtion to Call=> undefined
Part=> {
  id: 'chatcmpl-7jkPGZUX96bV5olCIFt8Zv9iKUw1h',
  object: 'chat.completion.chunk',
  created: 1691137502,
  model: 'gpt-3.5-turbo-0613',
  choices: [ { index: 0, delta: [Object], finish_reason: null } ]
}
Delta=> { function_call: { arguments: '}' } }
Content=> undefined
Inside Run Function
Calling Funtion Name=>
Funtion to Call=> undefined
Part=> {
  id: 'chatcmpl-7jkPGZUX96bV5olCIFt8Zv9iKUw1h',
  object: 'chat.completion.chunk',
  created: 1691137502,
  model: 'gpt-3.5-turbo-0613',
  choices: [ { index: 0, delta: {}, finish_reason: 'function_call' } ]
}
Delta=> {}
Content=> undefined

Before I invest that time, just want to clarify this is expected and will hopefully remain stable-ish before release?

Cheers

[OldManMeta]

I believe the docs are incorrect both here:
https://platform.openai.com/docs/guides/gpt/chat-completions-api
and
https://platform.openai.com/docs/guides/gpt/function-calling

This is more to do with the underlying model i.e. there is no "message" from the response any more, replaced with delta.

The second part on function calling is then where the confusion sets in - because it shows the following:

 # Step 4: send the info on the function call and function response to GPT
        messages.append(response_message)  # extend conversation with assistant's reply
        messages.append(
            {
                "role": "function",
                "name": function_name,
                "content": function_response,
            }
        )  # extend conversation with function response

The confusing thing about this is that as far as I can tell, when a function call is returned from GPT, the content field in the delta will always be null. So this comment in the code above "# extend conversation with assistant's reply" is the incorrect line.

[OldManMeta]

So I believe I have found a bug in GPT, and both your library and mine are suffering from the same thing.

If we make a call that results in a function call being streamed back, and then fully completed, everything works as expected.

If we make exactly the same call again immediately, the stream object itself is reporting that the stream done = true.

So this is what stops a second response straight away or at all. Effectively, from the GPT side, the stream has ended.

I checked in my wrapper, calling the api endpoints in streaming mode, and the same thing happens. I've checked across the board that I am calling res.end() on the stream, and not that I am calling reader.releaseLock(); as well.

This is my streaming code below after calling the openai api endpoint:

 const reader = response.body?.getReader();
        if (reader !== undefined) {

            while (true) {
                const { done, value } = await reader.read();

                if (done) {
                    // The stream has ended
                    console.log('Done called in Perform Call');                    
                    reader.releaseLock();                    
                    break;
                }

                // Convert the received data chunk to text
                const textChunk = convertChunkToText(value.toString());


                await assessParts(messageCollection, res, textChunk);

            }
        }

Where is the appropriate place for me to register this as a bug?

Cheers

[OldManMeta]

Just as an update here - I can confirm this is specific to function calling.

I can sending in varying prompts and get continuous replies correctly, but send in the same prompt that then requests a function call and the second time around, the stream will stop when the function response is sent back to GPT.

[rattrayalex]

If you can get a repro using curl sending the same request twice in this way, please report it to https://community.openai.com

However, I suspect there may be an application level bug here. Are you using NextJS? This could be related to their data cache, or another caching layer.

[rattrayalex]

Regarding the function-calling docs, note that they have messages.append(response_message) (where response_message will be something like {"role": "assistant", "content": "Sure, here is the weather in boston: "}) whereas in your code above it specifies "role": "user".

Note also that you don't want the content or function arguments from only that part of the stream, but from everything collected so far. This is how you can get a non-null content along with function call arguments.

We're working on helpers for these cases so it'll be easier :) hopefully you'll have them in the next couple of days!

[hiroshinishio]

Sorry if this has already been mentioned, but I couldn't find it, so let me ask you a question.Is there a callback function that can be triggered when a stream completes?

I develop and run a service called 'Q, ChatGPT for Slack' and I need to post to Slack with the text of the stream, but due to a strict rate limit I can only update once per second. Therefore, if we cannot clearly catch that the stream is completed, we risk missing the final update. We want to know clearly which triggers have been completed!

[rattrayalex]

Thank you @eliasm307 ! This makes a lot of sense, and I appreciate the detailed writeup.

Returning undefined to end the thread or a string (typically JSON.stringify(result), or possibly even an object to stringify automatically, sounds like it could work well.

We'd have to change the return type of the function if you used handler (or some equivalent) to be an array of ChatCompletions, at least if you're not also doing stream: true, so we'd likely only want to do this in a wrapping function. This could get confusing - we don't want our types to get too complicated.

What do you (or anyone else reading) think of something like this:

const chatCompletion = client.chat.completions.create({...params, stream: false}) // what we have today
const basicStream = client.chat.completions.create({...params, stream: true}) // what we have today

const fancyStream = client.chat.completions.stream(params)
fancyStream.on(…) // these methods only available if you use `.stream()`, which also auto-accumulates the response for you…
await fancyStream.finishedChatCompletion() // … to enable this kind of API for those that want it

const chatCompletions = client.chat.completions.runFunctions({...params, stream: false}, handlers) // with some way of passing handlers, perhaps the way you showed it

const chatCompletionsStream = client.chat.completions.runFunctionsStream(params, handlers) // a combination of the above two, name needs work
chatCompletionsStream.on(…)
await chatCompletionsStream.finishedChatCompletions()

[eliasm307]

We'd have to change the return type of the function if you used handler (or some equivalent) to be an array of ChatCompletions, at least if you're not also doing stream: true, so we'd likely only want to do this in a wrapping function

Hi @rattrayalex, do you have an example of what you mean above in terms of "so we'd likely only want to do this in a wrapping function"? Also can you clarify why the return has to be an array of ChatCompletions, would the model expect multiple completions for a single function call?

const fancyStream = client.chat.completions.stream(params)

Good idea making this into a separate stream method to show the difference in behaviour.

I can understand a separate stream method however I cant see the purpose of separate runFunctions and runFunctionsStream methods, what do these do that cant be done with stream? Is this just adding support for the handler functions? I think this could be confusing as it suggests the method will definitely result in a function call but the model either might choose not to call a function or might be forbidden from doing so based on other params e.g. setting function_call to none https://platform.openai.com/docs/api-reference/chat/create#chat/create-function_call.

[rattrayalex]

The return should be an array of ChatCompletions because the model may return multiple user-facing messages in the course of a single "conversation" (the model may choose to invoke functions multiple times, and they may be discontiguous). From the docs:

For example, if you ask the model “Find the weather in Boston this weekend, book dinner for two on Saturday, and update my calendar” and provide the corresponding functions for these queries, it may choose to call them back to back and only at the end create a user-facing message.

I'd hope for a single .runFunctions(…) wrapper function (or some equivalent) to run as many functions as the model requests.

It's possible we could have .stream() also run functions for you if you provide handlers somehow, and I could see how that'd be more convenient, but it'd be strange for this functionality to only be available if you're streaming.

[eliasm307]

@rattrayalex First of all, congratulations on the release of 4.0.0! This is a great improvement to the SDK!

The return should be an array of ChatCompletions because the model may return multiple user-facing messages in the course of a single "conversation" (the model may choose to invoke functions multiple times, and they may be discontiguous)

Ah yeah true, I was thinking you meant returning chat completions from the functions so I had made this sequence diagram of what I meant, but I get you meant the output of the completion creation call (still sharing the sequence diagram just to make sure we're on the same page). However, can I confirm this is just when stream: false? When there is a stream then these would just be parts of the stream?

the model may return multiple user-facing messages in the course of a single "conversation"

Also thats interesting, I had assumed a single request can only result in one "user-facing message". As a side note, if multiple discontiguous user-facing messages are possible then I think the helper methods suggested here #182 (reply in thread) (e.g. stream.contentDeltas, stream.contentSnapshots, or text = await stream.finishedContent()) wont work as they seem to assume its one user-facing message output, dont know if these were still under consideration.

I'd hope for a single .runFunctions(…) wrapper function (or some equivalent) to run as many functions as the model requests.

So just so we're on the same page, the purpose of .runFunctions would be to implicitly handle any function calls then if stream is false it only returns the first user-facing message (and if finish_reason is not "stop" then a new request can be done to get more user-facing responses?) and if stream is true does it stream the first user-facing message (and new requests need to be done to get more user-facing messages?) or is it meant to stream all the resulting user-facing messages while handling all the intermediate function calls?

It's possible we could have .stream() also run functions for you if you provide handlers somehow, and I could see how that'd be more convenient, but it'd be strange for this functionality to only be available if you're streaming.

The justification of "it'd be strange for this functionality to only be available if you're streaming" makes sense for needing new methods for this behaviour however I'm concerned about how this multiplies the number of available methods as it also includes combinations of other functionality e.g. runFunctionsStream, so if this same strategy is used for future functionality there could end up being a significant amount of options for creating completions.

Personally, I thought separate methods for streaming and non-streaming made sense however if this is done for combinations of other functionality then I think it's a slippery slope and might not be worth it in the long run. How about having these as create completion options e.g.

const chatCompletion = client.chat.completions.create({...params, stream: false}) // what we have today
const basicStream = client.chat.completions.create({...params, stream: true}) // what we have today

const fancyStream = client.chat.completions.create({...params, stream: true}, {autoAccumulate: true});
await fancyStream.finishedChatCompletion() // `autoAccumulate=true` enables this kind of API for those that want it

// this auto handles function calls and stops at the first user-facing message or gets all completions until finish reason "stop"?
const chatCompletions = client.chat.completions.create({...params, stream: false}, {functionCallHandlers: {...}}) 

const chatCompletionsStream = client.chat.completions.create({...params, stream: true}, {autoAccumulate: true, functionCallHandlers: {...}}) // a combination of accumulating user-facing messages content and auto handling function calls
chatCompletionsStream.on(…)
await chatCompletionsStream.finishedChatCompletions()

Having a separate argument for these options should make it easier to define type signatures to define what comes out with different combinations of options ie

• with no options then the current behaviour
• with autoAccumulate: true this only changes the type to a "fancy stream" if stream: true
• with functionCallHandlers defined and stream: false this means multiple completions are returned instead of one
• with functionCallHandlers defined and stream: true this doesn't change the output and maybe the behaviour of the stream also doesnt change so when looping over the parts with for await (...) when the code sees a function call it can just skip it as the handler will take care of it and the loop just needs to wait for the next user-facing message or the stream to finish
• with functionCallHandlers defined and autoAccumulate: true and stream: false then multiple completions are returned with function calls handled (autoAccumulate is redundant in this case)
• with functionCallHandlers defined and autoAccumulate: true and stream: true then a fancy stream is returned

This simplifies usage (I think) as it's one method and one place to configure the options for what is required, however, I can appreciate this could add some complexity to the implementation and/or maintenance.

Also regarding the stream.on(...) method, you mentioned this is only available on the "fancyStream", this is quite general so why not have this on the normal stream but have the events that can be listened to vary, e.g. events that require accumulation can only be used on a fancy stream but a normal stream can still have events like "functionCall", "finish", "error" etc

If the above is acceptable then the only difference between a normal and fancy stream is the fancy stream exposes other methods that rely on accumulation, so in this case would it be reasonable for this to actually be one stream that exposes all these methods, so there is just one type of stream (which also simplifies the signatures for chat.completions.create), where the fancyStream case automatically enables accumulation and the regular stream requires a manual call to accumulate or something and if the accumulate methods are used before this then an error is thrown with a message like You must call '.accumulate' on the stream after it is created to use this method.

However this could still cause bugs if people call accumulate at the wrong time (e.g. after streaming some or all the content), so what about having the accumulation behaviour driven from the chat.completions.create call via options so its always enabled during creation, so if accumulation is false or not set then the error says something like you must set "autoAccumulate" option to true when creating the stream?

[eliasm307]

Also from a UI app perspective an event to get the full accumulated messages or completions while streaming would be useful so this can be just passed to the UI component as is, e.g.

function ChatMessages() {
  const [chatMessages, setChatMessages] = useState([])
  ...
  function handleNewUserMessage(content) {
    ...
    const fancyStream = client.chat.completions.create(...); //  ie the current chat plus the new message
    // this would be updated while streaming in a message or when function calls are being processed etc, 
    // this reflects the state of the messages in realtime as the model is coming up with a response
    // the UI can then filter and/or format the messages for display etc
    fancyStream.on("messagesUpdated", (newMessages) => setChatMessages(newMessages.map(formatAiMessage))); 
  }

  return <>
    ..rendered chat messages
  </>
}

[AaronFriel]

This seems like it might be Azure OpenAI specific, but on a substantial number of requests, the initial promise awaiting the stream hangs, or during async iteration the object can be undefined:

logger.info(`🚀 Sending prompt of ${promptLength} tokens to OpenAI`);
const stream = await client.chat.completions.create({
  stream: true,
  messages: promptMessages,
  model: args.model,
});
// 💥 This log statement is never reached, the await hangs.
logger.info('Prompt sent');

const responseStream = new ReadableStream({
  async pull(controller) {
    let tokens = 0;
    for await (const chunk of stream) {
      // Sometimes `chunk.choices` is undefined, and we panic with:
      // error unhandledRejection: Error [TypeError]: Cannot read properties of undefined (reading '0')
      const content = chunk.choices[0].delta.content;
//                                 ^
// Error message points to exactly this line, indicating `chunk.choices` is undefined
// i.e.: chunk isn't a valid item object?
      controller.enqueue(encoder.encode(content));
    }
    controller.close();
    await cleanup();
  },
  async cancel() {
    stream.controller.abort();
    await cleanup();
  },
});

return new ai.StreamingTextResponse(responseStream, {
  headers: {
    // ...
  },
});

[rattrayalex]

Interesting…

1. I assume the hanging is a result of the Azure API – can you provide any tips on how to repro in a way that demonstrates it's a problem with this SDK?
2. Yes, Azure's API returns different types, and you'll need to use ?. there.

[OldManMeta]

I've raised this issue in here previously - it's and OpenAI API issue with their stream management. This occurs interacting with their rest services, and appears to be most often created when functions are provided. The good thing is that you can code around the issue and make your solution recognize the hang - the first retry appears to kick it back into action and the stream then functions correctly.

[rattrayalex]

The team is looking into this – if you can share any details of requests you attempted to send that were like this, it may be helpful. A request ID of a successful response just after or before an unsuccessful response with identical params may be handy if you have it.

[AaronFriel]

Anecdotally, these hangs appear more often with large prompts - I haven't yet measured but when sending a 2500 token prompt to an Azure OpenAI GPT-3.5 model, I notice issues almost every request.

The same payload sent to the GPT-4 model does not hang.

[rattrayalex]

If anyone is looking for a way to help – or just wants to migrate their codebase from v3.3.0 to v4.0.0 – we'd love for you to try using this tool to automatically migrate your repo!

You can try it locally now with these commands in the root of your project:

npx -y @getgrit/launcher apply openai_v4

(this will soon be made available as openai migrate)

Please share feedback, even if just to say that it worked well for you!

[eliasm307]

Is the link correct? I can't seem to open it on mobile

[rattrayalex]

My mistake; fixed!

[paras-malhotra]

I keep getting a connection reset error: cause: FetchError: request to https://api.openai.com/v1/audio/transcriptions failed, reason: read ECONNRESET on using the openai.audio.transcriptions.create method. Openai status page shows all green, so not sure why this is happening.

[rattrayalex]

Does this happen on other endpoints, or from curl on the same machine? Where are you making the request from? (bear in mind OpenAI's supported countries)

[paras-malhotra]

Thanks, the curl command really helped! It displayed the correct error (quota exceeded). Once fixed, it was working great.

[zhujiageng]

when i use "openai": "4.0.0-beta.11", sometimes see the error log in server: Cannot read properties of undefined (reading 'stream'), but i can't reproduct the error in my local computer.

[RobertCraigie]

Thanks for the report, do you have a stack trace for the error you're seeing?

[zhujiageng]

TypeError: Cannot read properties of undefined (reading 'stream') at defaultParseResponse (/app/node_modules/openai/core.js:87:21) at process.processTicksAndRejections (node:internal/process/task_queues:95:5)

[RobertCraigie]

Thanks! Would you be able to share a code snippet as well?

[RobertCraigie]

We believe we've identified the cause of this and will have a fix out shortly! Thanks for the bug report!

[rattrayalex]

The fix for this has been released as part of the 4.0.0 release!

[schnerd]

Thanks everybody for all the feedback on v4 of the SDK, we're super excited with how it turned out and your input was super helpful.

Please check out the v3 -> v4 migration guide to get started with v4.

[ipbuzzofficial]

(https://github.com/openai/openai-node/assets/136312930/bbddfb32-843d-40bc-9452-f8f279858932)
I'm getting this error

[rattrayalex]

As below, you want to import and use the type differently.

[ipbuzzofficial]

"use client";

import axios from "axios";
import * as z from "zod";
import { Heading } from "@/components/heading";
import { MessageSquare } from "lucide-react";
import { useForm } from "react-hook-form";
import { formSchema } from "./constants";
import { zodResolver } from "@hookform/resolvers/zod";
import { Form,FormField, FormItem,FormControl } from "@/components/ui/form";
import { Input } from "@/components/ui/input";
import { Button } from "@/components/ui/button";
import { useRouter } from "next/navigation";
import { useState } from "react";
import { ChatCompletionMessage } from "openai";
import type OpenAI from "openai";

const ConversationPage = () => {
const router = useRouter();
const [messages, setMessages] = useState<ChatCompletionMessage[]>([]);

const form = useForm<z.infer<typeof formSchema>>({
    resolver: zodResolver(formSchema),
    defaultValues: {
        prompt: ""
    }
});

 const isLoading = form.formState.isSubmitting;

 const onSubmit = async (values: z.infer<typeof formSchema>) => {
    try {
        const userMessage:  OpenAI.Chat.ChatCompletionMessage = {role: "user", content: values.prompt};
        
        const newMessages = [...messages, userMessage];

        const response = await axios.post("/api/conversation", {
            messages: newMessages,

        });

        setMessages((current) => [...current, userMessage, response.data]);
        form.reset();


    } catch(error:any) {
        // TODO: Open Pro Modal
        console.log(error);
    } finally {
        router.refresh();
    }
 };


return(
    <div>
   <Heading 
   title="Conversation"
   description="Our most advanced conversation model."
   icon={MessageSquare}
   iconColor="text-violet-500"
   bgColor="bg-violet-500/10"/>

    <div className="px-4 lg:px-8">
        <div>
            <Form{...form}>
            <form
            onSubmit={form.handleSubmit(onSubmit)}
            className="rounded-lg
            border
            w-full
            p-4
            px-3
            md:px-6
            focus-within:shadow-sm
            grid
            grid-cols-12
            gap-2">
                <FormField 
                name="promt"
                render={({field}) => (
                    <FormItem className="col-span-12 lg:col-span-10 ">
                        <FormControl className="m-0 p-0">
                            <Input 
                            className="border-0 outline-none focu-visible:ring-0 focu-visible:ring-transparent" disabled={isLoading}
                            placeholder="How do I calculate the radius of a circle?" {...field}
                            />
                        </FormControl>
                    </FormItem>
                )}
                
                />
                <Button className="col-span-12 lg:col-span-2 w-full" disabled={isLoading}>
                        Generate

                </Button>
            </form>
            </Form>

        </div>
        <div className="space-y-4 mt-4">
                    <div className="flex flex-col-reverse gap-y-4">
                        {messages.map((message) => (
                            <div 
                            key={message.content}>

                            {message.content}
                            </div>
                        ))}
                    </div>
        </div>
    </div>


</div>
)

}
export default ConversationPage;

Please help me with this code I'm getting error at import

[rattrayalex]

Instead of import { ChatCompletionMessage } from "openai" you want

import OpenAI from "openai";

type ChatCompletionMessage = OpenAI.Chat.ChatCompletionMessage

[theogravity]

What about ChatCompletionRequestMessageRoleEnum ?

[rattrayalex]

I think you're looking for ChatCompletionRole

[a2thek26]

I'm loading a couple pieces like this in my frontend, mostly to use .fromReadableStream. It works when in dev mode (nextjs) but when I build I get an error about Audio already being declared in the app console. I'm not using anything with "audio" so I'm going to guess it's the way I'm loading them that's either causing it to load twice or there's something else going on.

import { APIUserAbortError } from 'openai';
import { ChatCompletionStream } from 'openai/lib/ChatCompletionStream';

I'm loading the abort error to handle when the stream gets aborted.

[rattrayalex]

Can you report this as an issue instead?

(as a workaround, you can try importing the error class directly from the file it's declared in, or just checking error.name === 'APIUserAbortError').

[kenjam97]

Also seeing this same error when I build (nextjs), no issues in dev mode at all, only in production

[rattrayalex]

Moved to this issue: #552