Neocortex Unity SDK

The Neocortex Unity SDK is a Unity package that allows you to easily integrate Neocortex into your Unity project. The SDK provides a set of APIs that allow you to interact with the projects created on the Neocortex web platform.

Requirements

• Neocortex account Sign Up Here
• Unity 2021.3 or above Download Here
• Git version control system Download Here

Installation

• Open your Unity project
• Go to Window > Package Manager
• Click on the + button and select Add package from git URL
• Paste the following URL: https://github.com/neocortex-link/neocortex-unity-sdk.git
• Click on the Add button

Setup

Save API Key

To start using the Neocortex SDK, you need to initialize it with your Neocortex API key. You can create a new API key from the Neocortex web platform by going to the API Keys page.

• Create a new API key and copy it
• Open your Unity project
• Go to Tools > Neocortex Settings
• Paste the API key in the API Key field and click on the Save button

Create a new Neocortex project

• Go to the Neocortex web platform and create a new project
• Copy the project ID from the project details page
• Open your Unity project and go to your scene
• Create an empty GameObject and add the Neocortex Smart Agent component to it
• Paste the project ID in the Project ID field

API Reference

After setting up the Neocortex SDK in your Unity project, you can start using the APIs to interact with the Neocortex project.

Neocortex Smart Agent component

The Neocortex Smart Agent component is the main component that allows you to interact with the Neocortex project.

public async void TextToText(string message)

• Send a text message to the Neocortex project, and expect a text response.
• Parameters:
  • message: The text message to send.
• Example:

  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnChatResponseReceived.AddListener((response) =>
  {
      Debug.Log($"Message: {response.message}");
      Debug.Log($"Action: {response.action}");
  });
  smartAgent.TextToText("Hello, Neocortex!");

public async void TextToAudio(string message)

• Send a text message to the Neocortex project, and expect a audio response.
• Parameters:
  • message: The text message to send.
• Example:

  var audioSource = GetComponent<AudioSource>();
  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnChatResponseReceived.AddListener((response) =>
  {
      Debug.Log($"Message: {response.message}");
      Debug.Log($"Action: {response.action}");
  });
  smartAgent.OnAudioResponseReceived.AddListener((audioClip) =>
  {
      audioSource.clip = audioClip;
      audioSource.Play();
  });
  
  smartAgent.TextToAudio("Hello, Neocortex!");

public async void AudioToText(AudioClip audio)

• Sends an audio clip to the Neocortex project. This method is used with NeocortexAudioReceiver component to send audio data.
• Parameters:
  • audioClip: The audio clip to send.
• Example:

  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnTranscriptionReceived.AddListener((message) =>
  {
      Debug.Log($"You: {message}");
  });
  
  var audioReceiver = GetComponent<NeocortexAudioReceiver>();
  audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
  {
      Debug.Log($"Audio Data Length: {audioClip.samples}");
      smartAgent.AudioToText(audioClip);
  });
  
  // Start recording audio for 3 seconds
  audioReceiver.StartMicrophone();
  await Task.Delay(3000);
  audioReceiver.StopMicrophone();

public async void AudioToAudio(AudioClip audio)

• Sends an audio clip to the Neocortex project and expects an audio response. This method is used with NeocortexAudioReceiver component to send audio data.
• Parameters:
  • audioClip: The audio clip to send.
• Example:

  var audioSource = GetComponent<AudioSource>();
  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnAudioResponseReceived.AddListener((audioClip) =>
  {
      audioSource.clip = audioClip;
      audioSource.Play();
  });
  smartAgent.OnTranscriptionReceived.AddListener((message) =>
  {
      Debug.Log($"You: {message}");
  });
  smartAgent.OnChatResponseReceived.AddListener((response) =>
  {
      Debug.Log($"Message: {response.message}");
      Debug.Log($"Action: {response.action}");
  });
  
  var audioReceiver = GetComponent<NeocortexAudioReceiver>();
  audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
  {
      Debug.Log($"Audio Data Length: {audioClip.samples}");
      smartAgent.AudioToAudio(audioClip);
  });
  
  // Start recording audio for 3 seconds
  audioReceiver.StartMicrophone();
  await Task.Delay(3000);
  audioReceiver.StopMicrophone();

public UnityEvent OnChatResponseReceived

• Event that is triggered when the Neocortex project responds to a text message.
• Parameters:
  • response: The response from the Neocortex project.
• Example:

  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnChatResponseReceived += (response) =>
  {
      Debug.Log($"Message: {response.message}");
      Debug.Log($"Action: {response.action}");
  };

public UnityEvent OnTranscriptionReceived

• Event that is triggered when the Neocortex project transcribes an audio message to text.
• Parameters:
  • message: The transcribed audio message.
• Example:

  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnTranscriptionReceived += (message) =>
  {
      Debug.Log($"You: {message}");
  };

public UnityEvent OnAudioResponseReceived

• Event that is triggered when the Neocortex project responds with an audio message.
• Parameters:
  • audioClip: The audio clip received from the Neocortex project.
• Example:

  var audioSource = GetComponent<AudioSource>();
  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnAudioResponseReceived += (audioClip) =>
  {
      audioSource.clip = audioClip;
      audioSource.Play();
  };

public UnityEvent OnRequestFailed

• Event that is triggered when a request to the Neocortex project fails.
• Parameters:
  • error: The error message.
• Example:

  var smartAgent = GetComponent<NeocortexSmartAgent>();
  smartAgent.OnRequestFailed += (error) =>
  {
      Debug.LogError(error);
  };

NeocortexAudioReceiver component

The NeocortexAudioReceiver component is used to record audio data from the microphone via loudness of the souned, so you can have a hands free chat with the smart agent. On this component you can:

• pick the microphone device to use
• set the amplitude threshold for when to start and stop recording
• set the max wait time for the recording to automatically stop if no sound is detected

public void StartMicrophone()

• Starts recording audio from the microphone
• Example:

var audioReceiver = GetComponent<NeocortexAudioReceiver>();
audioReceiver.StartMicrophone();

public void StopMicrophone()

• Stops recording audio from the microphone
• Example:

var audioReceiver = GetComponent<NeocortexAudioReceiver>();
audioReceiver.StopMicrophone();

public UnityEvent OnAudioRecorded OnAudioRecorded

• Event that is triggered when audio data is recorded from the microphone.
• Returns:
  • audioClip: The recorded audio clip.
• Example:

var audioReceiver = GetComponent<NeocortexAudioReceiver>();
audioReceiver.OnAudioRecorded.AddListener((audioClip) =>
{
    Debug.Log($"Audio Data Length: {audioClip.samples}");
});

Sample Projects

You can find sample projects that demonstrate how to use the Neocortex Unity SDK in the Package Manager window under the Samples section of the Neocortex package.