Install

Get the asset from the Unity Asset Store here: https://assetstore.unity.com/packages/tools/generative-ai/ai-commander-281306

Import The Asset

Import the asset from Unity's Package Manager.

Press Import

Import the Newtonsoft Package using Unity's Package Manager

Open the package manager (Windows -> Package Manager)

Open the package manager

Click the plus icon on the upper left-hand corner, then Add package by name.

Add a package by name

Fill in the package details:

Package Name: com.unity.nuget.newtonsoft-json

Version: 3.2.1

Click Add

Click the Add button.

Get An Open AI API Key

Create a developer account with Open AI, and get an API key here: https://platform.openai.com/api-keys

Download The Server Project

Clone the Server Project from this GitHub repo: https://github.com/Danejw/ai-python-server

Open the repo in VS Code or by using an IDE of your choice.

1. Press Ctrl+Shift+P (or Cmd+Shift+P on macOS) to open the Command Palette.

2. Type "Python: Select Interpreter" and press Enter.

3. A list of available Python interpreters will appear.

4. Find the interpreter corresponding to Python 3.11.8 and select it.

Import all the server dependencies.

1. Open the integrated terminal by pressing Ctrl+` (backtick) or by navigating to View -> Terminal -> New Terminal from the top menu.

2. Once you're in the correct directory, run the following command to install all the dependencies:

pip install -r requirements.txt

Store your Open AI API key in the .env variable of the server project.

Create a file called .env with the contents of you OPENAI_API_KEY, like so:

(replacing your-api-key with the OpenAI API Key provided by Open AI)

Find the .env file in the project and paste your api key in place of the 'your-api-key', then save your project.

Run the uvicorn command in a terminal:

 uvicorn api:app --reload

Visit http://127.0.0.1:8000/docs, to view your Fast API locally running server.

Play The Demo Scene In Unity

While the server is running, return to Unity. Play the demo scene, and interact with the demo scene by recording your voice or typing in a prompt and clicking send.

Function Calling -> Scenes -> Fetch Demo