WhisperGPT Repo for CS639: Capstone Fall2023

The whisperGPT folder contains the program that implements a program that is able to talk to the user. It uses GPT-4, Whisper-1, and TTS-1 models from OpenAI API. The Unity version is 2022.3.10f1.

The program in this folder is used for develop and test the whisperGPT foler, and will be integrated into the VR program here

Little Demo

Video demo

Photo of WhisperGPT UI:

In the VR program:

Availability

I used the OpenAI-Unity package to implement the functions. However, since the OpenAI is constantly updating their API, I changed something in the package for multiple times (The lastest update: 12/15/2023):

In Packages/com.srcnalt.openai-unity/Runtime/DataTypes.cs, I added the following parameter to "public struct CreateChatCompletionResponse : IResponse"

public string system_fingerprint { get; set; }

And I added the following parameter to "public struct ChatChoice"

public bool logprobs { get; set; }

Also, the package didn't provide TTS model. Therefore, it is recommended to download the files at here rather than cloning the repo.

Setting Up Your OpenAI Account

To use the OpenAI API, you need to have an OpenAI account. Follow these steps to create an account and generate an API key:

• Go to https://openai.com/api and sign up for an account
• Once you have created an account, go to https://beta.openai.com/account/api-keys
• Create a new secret key and save it

Saving Your Credentials

To make requests to the OpenAI API, you need to use your API key and organization name (if applicable). To avoid exposing your API key in your Unity project, you can save it in your device's local storage.

To do this, follow these steps:

• Create a folder called .openai in your home directory (e.g. C:User\UserName\ for Windows or ~\ for Linux or Mac)
• Create a file called auth.json in the .openai folder
• Add an api_key field to the auth.json file and save it
• Here is an example of what your auth.json file should look like:

{
    "api_key": "sk-...W6yi"
}

IMPORTANT: Your API key is a secret. Do not share it with others or expose it in any client-side code (e.g. browsers, apps). If you are using OpenAI for production, make sure to run it on the server side, where your API key can be securely loaded from an environment variable or key management service.