Core Concepts

Understand the architecture and the main classes of the Estuary Unreal Engine SDK.

Architecture

The SDK is a single Unreal plugin with one always-on core and three optional modules:

Modules

Module	What it does	Platforms
Estuary	Core runtime: connection, text, voice, TTS, lipsync, vision. Always compiles.	Win64 / Mac / iOS / Android
EstuaryLiveKit	Optional WebRTC voice with hardware-grade echo cancellation. Auto-detected.	Win64 / Mac
EstuaryLiveLink	Live Link source that delivers ARKit-52 blendshapes to your MetaHuman.	All four
EstuaryEditor	Editor tooling; shows a startup notification when the optional LiveKit prebuilts are absent.	Win64 / Mac

When the optional LiveKit binaries are absent, the SDK still compiles and runs — voice transparently falls back to the WebSocket transport.

---

UEstuarySubsystem (the connection)

UEstuarySubsystem is a UGameInstanceSubsystem — there is exactly one per Game Instance, created automatically. It owns the single Socket.IO session and is the entry point for connecting.

Access it from anywhere:

// C++
UEstuarySubsystem* Estuary = GetGameInstance()->GetSubsystem<UEstuarySubsystem>();

In Blueprint, use Get Estuary Subsystem (under Estuary → Helpers) or Get Game Instance → Get Subsystem with the class pin set to EstuarySubsystem.

The subsystem:

1. Opens and authenticates the connection (Connect / Disconnect).
2. Drives the session state machine and broadcasts OnStateChanged.
3. Receives every server event and routes it to the correct character component.
4. Exposes session-wide events: OnSessionConnected, OnAuthError, OnSessionRejected, OnError, OnQuotaExceeded, OnMemoryUpdated.

:::note One session at a time A subsystem holds a single authenticated session scoped to one character ID. You can place multiple AEstuaryCharacter actors in a level and they coexist cleanly, but the active conversation is with the character ID used in Connect. :::

---

AEstuaryCharacter (the actor)

AEstuaryCharacter is a Blueprintable actor you place in your level. It is a thin container — set its Character ID and it spawns the sub-components that do the work:

Sub-component	Role
UEstuaryCharacterComponent	Send text, interrupt, update preferences; fires per-character response events
UEstuaryMicrophoneComponent	Captures the player's microphone for voice input
UEstuaryAnimationReceiver	Applies the incoming ARKit-52 lipsync stream to the MetaHuman face
UEstuaryUserPositionComponent	(Body animation) streams the player's floor position for conditioning

You add UEstuaryAudioPlayerComponent (TTS playback) and UEstuaryCameraComponent (vision) yourself when you need them.

To customize, create a Blueprint subclass of AEstuaryCharacter and add it to your MetaHuman actor, or merge the components onto an existing character.

---

UEstuaryCharacterComponent (the conversation)

This is the component you talk to most. It exposes the conversation API and the per-character event delegates:

// Send a user message — returns the message_id for correlation
FString MessageId = CharacterComponent->SendText(TEXT("What's the weather like?"));

// Script a prewritten line (TTS only, skips the LLM)
CharacterComponent->SayLine(TEXT("Welcome back!"), /*bTextOnly=*/false);

// Interrupt the in-flight response
CharacterComponent->Interrupt();

Bind these events (in Blueprint as On ... nodes, or in C++ with AddDynamic):

Event	Fires when
OnBotResponse	Each streaming text chunk arrives (bIsFinal true on the last)
OnBotVoice	Each streaming TTS audio chunk arrives
OnSttResponse	A partial or final speech-to-text result arrives during voice input
OnInterruptAcked	The server confirms an interrupt
OnBotAnimation	A renderable ARKit-52 lipsync frame arrives
OnVisionResponse	A vision (VLM) result for a sent camera frame arrives
OnCameraCaptureRequested	The server proactively asks the client to capture an image

---

Session Lifecycle

The subsystem exposes its state through the EEstuarySessionState enum and the OnStateChanged event:

State	Meaning
Disconnected	No active socket. Initial and post-Disconnect state.
Connecting	Socket opening and authenticating.
Connected	Session live — ready for text, voice, and animation.
Reconnecting	Transport dropped unexpectedly; backing off and retrying.
Error	Terminal failure (bad credentials or policy rejection). Call Connect again to recover.

Reconnection uses exponential backoff, capped by Reconnect Max Backoff (seconds) in Project Settings.

---

Transport

The SDK connects over Socket.IO v4 using the WebSocket transport only. The connection is authenticated with an auth payload carrying your API key, character ID, and player ID.

Voice can run over one of two transports:

• WebSocket (default, all platforms): audio chunks stream over the same Socket.IO connection. Use half-duplex push-to-talk to avoid the character hearing itself.
• LiveKit WebRTC (Win64 / Mac): lower-latency full-duplex audio with built-in acoustic echo cancellation. Auto-enabled when the LiveKit binaries are present, unless you set Disable LiveKit in Project Settings.

See Voice Connection for details.

---

Blueprint vs C++

Everything is reachable from both. The public API is exposed as:

• BlueprintCallable functions on the actors, components, and subsystem.
• Latent async nodes for long-running operations (Connect (Latent), Acquire Microphone Permission).
• BlueprintAssignable delegates for every event (On Bot Response, On STT Response, …).
• Static helpers in UEstuaryBlueprintLibrary (message IDs, base64 PCM codecs, auth payload factories).

C++ users call the same methods directly and bind delegates with AddDynamic.

---

Next Steps

• Text Chat — Streaming text and interrupts
• Voice Connection — Microphone, TTS, and LiveKit
• Animation & Lipsync — MetaHuman face animation
• API Reference — Full class documentation