Skip to main content

Getting Started

Set up the Estuary Unreal Engine SDK and run your first AI conversation with a MetaHuman in under 15 minutes.

Prerequisites​

RequirementDetails
Unreal Engine5.6 or newer
IDEVisual Studio 2022 with the Game development with C++ workload (Windows), or Xcode (macOS)
gitOn your PATH — used to clone the SDK and its dependency plugins
Estuary AccountSign up at app.estuary-ai.com
API KeyGenerated from your Estuary dashboard (est_... format)
Character IDUUID of the AI character you want to use

The SDK is a C++ plugin with a Blueprint-first API — you can build a complete voice conversation without writing any C++. A C++-capable project is still required because the plugin compiles from source.

Platform Support​

FeatureWin64MaciOSAndroid
Text & voice conversation (Socket.IO / WebSocket)✅✅✅✅
Microphone capture✅✅✅✅
Streaming TTS playback✅✅✅✅
MetaHuman ARKit-52 lipsync✅✅✅✅
Scene / webcam vision capture✅✅✅✅
LiveKit WebRTC voice (with echo cancellation)✅✅——
Editor tools✅✅——

On iOS and Android, voice runs over the WebSocket transport. LiveKit WebRTC is desktop-only.

Installation​

The SDK lives on GitHub at github.com/estuary-ai/estuary-unreal-engine-sdk. It ships as an Unreal plugin you drop into your project's Plugins/ folder.

# From your UE 5.6+ project root
cd Plugins
git clone https://github.com/estuary-ai/estuary-unreal-engine-sdk Estuary

Then:

  1. Right-click your .uproject → Generate Visual Studio project files.
  2. Open the project. Unreal will offer to build the plugin modules — click Yes.

Dependencies​

The SDK depends on two third-party plugins. They are optional at compile time — the SDK compiles in a stub mode without them (connection, voice, and mic become no-ops) so the editor always opens — but you need both for full functionality. Clone them into Plugins/ alongside the SDK:

# From your project's Plugins/ folder

# SocketIOClient vendors asio, rapidjson, and websocketpp as git submodules.
# --recurse-submodules is REQUIRED — without it the build fails with
# "Cannot open include file: 'asio/system_timer.hpp'".
git clone --depth 1 --branch v2.11.0 --recurse-submodules https://github.com/getnamo/SocketIOClient-Unreal SocketIOClient

# Runtime Audio Importer (optional — read the caution below before using this OSS clone)
git clone --depth 1 https://github.com/gtreshchev/RuntimeAudioImporter RuntimeAudioImporter

:::caution SocketIOClient submodules are mandatory If you already cloned SocketIOClient without --recurse-submodules (empty Source/ThirdParty/asio, rapidjson, websocketpp folders), pull them in before building:

cd SocketIOClient
git submodule update --init --recursive

:::

DependencyPurposeRequired for
SocketIOClient (getnamo, v2.11.0)Socket.IO v4 / WebSocket transportAll networking — without it, connection and voice are no-ops
Runtime Audio ImporterCross-platform microphone captureVoice input on Mac, iOS, and Android (Win64 has a built-in mic fallback)

After cloning both plugins, regenerate Visual Studio project files and rebuild. With both present, the SDK defines WITH_ESTUARY_SOCKETIO=1 and WITH_ESTUARY_RAI=1 and enables full functionality automatically.

:::caution Runtime Audio Importer compatibility The OSS main of Runtime Audio Importer can drift from the version this SDK targets. A fresh clone of main may fail to compile against recent UE versions (5.6–5.8) — e.g. minimp3.h selecting an ARM-NEON code path on MSVC ('float32_t': undeclared identifier), or 'OnAudioChunkReceived': is not a member of 'UCapturableSoundWave' if the capture API has changed upstream. If you hit either, use the Fab edition of Runtime Audio Importer instead (delete the OSS clone first), which is the supported path for production.

Runtime Audio Importer is optional: on Win64 the SDK falls back to a built-in microphone capture path, so you can omit this plugin entirely and still use voice on desktop. It is only required for mic capture on Mac, iOS, and Android. :::

Configure Project Settings​

Open Project Settings → Plugins → Estuary and set:

FieldValue
Server URLhttps://api.estuary-ai.com (or your self-hosted gateway)
Audio Sample Rate24000 (default)

Then open Project Settings → Plugins → Estuary Secret and set:

FieldValue
Auth TokenYour est_... API key

:::info Where the API key is stored The key is written to Config/DefaultEstuarySecret.ini, a local-only file that is git/Perforce-ignored — it never gets committed. For shipping builds, don't bake a key into config; pass it at runtime via the Connect node's auth payload instead. :::

Add a Character to Your Level​

  1. In the Place Actors panel, search for Estuary Character and drag it into your level (or create a Blueprint subclass of AEstuaryCharacter).
  2. Select the actor and set Character ID to your character's UUID from the dashboard.

AEstuaryCharacter automatically creates its conversation, microphone, and animation sub-components — you don't add them manually.

Connect and Send Your First Message​

Open the Level Blueprint and build this graph on Event BeginPlay:

  1. Add a Make Auth Payload from Settings node (under Estuary → Helpers). Pass your character's Character ID and a Player ID (any opaque per-user string). It returns an auth payload and the configured server URL.
  2. Add a Connect (Latent) node (under Estuary → Transport). Feed in the Server URL and Auth payload from the previous node.
  3. From the On Connected output pin, get your character actor's Character Component and bind its On Bot Response event.
  4. Still on On Connected, call Send Text on the Character Component with a message like "Hello! Tell me a fun fact."
Event BeginPlay
  → Make Auth Payload from Settings (CharacterId, PlayerId) ─┬─ Auth ──┐
                                                             └─ ServerUrl ─┐
  → Connect (Latent) [ServerUrl, Auth]
        ├─ On Connected ──→ Bind Event to (CharacterComponent.On Bot Response)
        │               └─→ CharacterComponent.Send Text ("Hello! Tell me a fun fact.")
        ├─ On Auth Failed ──→ Print "Auth failed"
        ├─ On Rejected ─────→ Print "Session rejected"
        └─ On Network Error ─→ Print error

In your On Bot Response handler, check Is Final and print Text to see the character's reply in the log.

:::note Send Text vs. Say Line Use Send Text to send a user message and get the character's LLM-generated reply. Say Line is different — it makes the character speak prewritten text verbatim via TTS and skips the LLM, so it won't produce a conversational answer. :::

The same flow in C++​

void AMyGameMode::BeginPlay()
{
    Super::BeginPlay();

    UEstuarySubsystem* Estuary = GetGameInstance()->GetSubsystem<UEstuarySubsystem>();

    FEstuaryAuthPayload Auth;
    Auth.ApiKey      = TEXT("est_..."); // or leave empty to use Project Settings
    Auth.CharacterId = TEXT("your-character-uuid");
    Auth.PlayerId    = TEXT("player-123");

    Estuary->OnSessionConnected.AddDynamic(this, &AMyGameMode::HandleConnected);
    Estuary->Connect(TEXT("https://api.estuary-ai.com"), Auth);
}

void AMyGameMode::HandleConnected(const FEstuarySessionInfo& Session)
{
    UE_LOG(LogTemp, Log, TEXT("Connected! Session: %s"), *Session.SessionId);
}

Add Voice (Push-to-Talk)​

AEstuaryCharacter already includes a microphone component. To enable streaming TTS playback, add an audio player and wire a push-to-talk button:

  1. On your character actor (or its Blueprint), add an Estuary Audio Player Component. It auto-wires to the character's conversation stream and plays incoming TTS.
  2. In your input mapping, bind a key (e.g., V):
    • Pressed → call Start Listening on the character's Microphone Component.
    • Released → call Stop Listening.

Press Play in Editor, hold the key, speak, and release. You'll see live transcription via On STT Response and hear the reply through the audio player.

For MetaHuman lipsync, see Animation & Lipsync.

Verify It Works​

In the Output Log, filter for LogEstuary. A healthy text session logs (component/subsystem prefixes are part of the real output — filter on them if needed):

LogEstuary: [Subsystem] Connect ENTER — ServerUrl=https://api.estuary-ai.com state=0
LogEstuary: [Bridge] authenticate emitted char=... player=... rate=24000 anim=false body_anim=false api_key=est_***
LogEstuary: [Subsystem] session_info received: session_id=... conversation_id=... char=... player=...
LogEstuary: [EstuaryCharacterComponent] SendText '...' (message_id=...)

Adding voice also logs [EstuaryMicrophoneComponent] capture lines and bot_voice playback. The bot's reply itself routes silently to the Character Component's On Bot Response event — bind it (and print Text) to see the reply in the log. If you see Warning: [Subsystem] HandleBotResponse: no registered component for CharacterId='...', the actor's Character ID doesn't match the ID in the Connect auth payload — set them to the same UUID.

Next Steps​

Last updated