Install

Get the asset from the Unity Asset Store here: https://assetstore.unity.com/packages/tools/generative-ai/ai-commander-281306

Import The Asset

Import the asset from Unity's Package Manager.

Import the Newtonsoft Package using Unity's Package Manager

Open the package manager (Windows -> Package Manager)

Click the plus icon on the upper left-hand corner, then Add package by name.

Fill in the package details:

Package Name: com.unity.nuget.newtonsoft-json

Version: 3.2.1

Click the Add button.

Get An Open AI API Key

Create a developer account with Open AI, and get an API key here: https://platform.openai.com/api-keys

Download The Server Project

Clone the Server Project from this GitHub repo: https://github.com/Danejw/ai-python-server

Open the repo in VS Code or by using an IDE of your choice.

Press Ctrl+Shift+P (or Cmd+Shift+P on macOS) to open the Command Palette.
Type "Python: Select Interpreter" and press Enter.
A list of available Python interpreters will appear.
Find the interpreter corresponding to Python 3.11.8 and select it.

Import all the server dependencies.

Open the integrated terminal by pressing Ctrl+` (backtick) or by navigating to View -> Terminal -> New Terminal from the top menu.
Once you're in the correct directory, run the following command to install all the dependencies:

pip install -r requirements.txt

Store your Open AI API ke y in the .env variable of the server project.

Create a file called .env with the contents of you OPENAI_API_KEY, like so:

(replacing your-api-key with the OpenAI API Key provided by Open AI)

Run the uvicorn command in a terminal:

 uvicorn api:app --reload

Play The Demo Scene In Unity

While the server is running, return to Unity. Play the demo scene, and interact with the demo scene by recording your voice or typing in a prompt and clicking send.