Skip to main content

Voice Connection

Add real-time speech: capture the player's microphone, stream it for transcription, and play the character's streaming TTS reply.

Overview​

A voice turn has three parts:

  1. Microphone capture — UEstuaryMicrophoneComponent records the player and streams audio to the server.
  2. Transcription — the server returns partial and final speech-to-text via OnSttResponse.
  3. TTS playback — UEstuaryAudioPlayerComponent plays the character's spoken reply (bot_voice) with pop-free streaming.

Voice runs over one of two transports — WebSocket (all platforms) or LiveKit WebRTC (desktop).

Setup​

AEstuaryCharacter already includes a UEstuaryMicrophoneComponent. Add an audio player for playback:

  1. On your character actor, Add Component → Estuary Audio Player Component. It auto-discovers the sibling conversation component and plays incoming TTS automatically.
  2. (Mobile) Request microphone permission before capturing — see Permissions.

Push-to-Talk​

The simplest and most robust pattern. Capture only while a key or button is held.

// Bind to an input action
void AMyPawn::OnTalkPressed()
{
    MicrophoneComponent->StartListening();
}

void AMyPawn::OnTalkReleased()
{
    MicrophoneComponent->StopListening();
}

In Blueprint, wire your input action's Pressed to Start Listening and Released to Stop Listening on the character's Microphone Component.

StartListening begins the voice session and streams microphone audio; StopListening ends it and triggers final transcription, after which the bot's text and voice reply stream back.

Live Transcription​

Bind On STT Response to show what the player is saying as they speak:

MicrophoneComponent->GetOwner()
    ->FindComponentByClass<UEstuaryCharacterComponent>()
    ->OnSttResponse.AddDynamic(this, &AMyPawn::HandleTranscript);

void AMyPawn::HandleTranscript(const FEstuarySttResponse& Stt)
{
    SubtitleText = Stt.Text;
    if (Stt.bIsFinal)
    {
        // Final transcript — the bot will now respond
    }
}

Microphone capture is fixed at 16 kHz mono for transcription, regardless of your TTS sample-rate setting.

Microphone Controls​

UEstuaryMicrophoneComponent exposes:

Method / PropertyDescription
StartListening() / StopListening()Open / close the push-to-talk window.
SetMuted(bool)User-controlled mute. Never toggled automatically.
SetSuppressed(bool)Half-duplex guard — temporarily stop sending while the bot speaks (WebSocket voice).
SetAutoBargeIn(bool)Enable/disable automatic interruption when the user starts speaking. Default on.
bIsCapturingTrue between StartListening and StopListening.
bIsMuted / bIsSuppressedCurrent mute / suppression state.

Barge-in (interrupting by speaking)​

With Auto Barge-In enabled (the default), if the player starts speaking while the character is talking, the SDK automatically interrupts the bot so the player can take over. Disable it with SetAutoBargeIn(false) if you prefer strict turn-taking.

TTS Playback​

UEstuaryAudioPlayerComponent is a UAudioComponent subclass that decodes streaming bot_voice audio and plays it through Unreal's audio engine with a small pre-buffer to avoid pops.

It auto-wires on BeginPlay — no manual binding needed. Useful members:

MethodDescription
GetCurrentAudioTime()Seconds elapsed in the current utterance (used by lipsync timing).
IsBotAudioActive()True while TTS is playing.
FlushAllForMessage(MessageId)Drop queued audio for a message (used on interrupt).
EnqueueRawPcm16(Bytes, SampleRate)Manually feed PCM if you build custom audio.

The default TTS playback rate is 24 kHz. Change it via Audio Sample Rate in Project Settings (the value is sent to the server at connect time). A reconnect is required for a new rate to take effect.

Voice Transports​

WebSocket voice (default, all platforms)​

Audio streams over the same Socket.IO connection as text. There is no acoustic echo cancellation, so use push-to-talk or the SetSuppressed half-duplex guard to keep the character from hearing its own TTS through an open microphone.

LiveKit WebRTC voice (Win64 / Mac)​

LiveKit provides lower-latency, full-duplex audio with built-in acoustic echo cancellation — the player can speak naturally while the character talks, hands-free, without echo.

To enable LiveKit voice:

  1. Run the one-time prebuilt fetcher per workstation:
    • Windows: powershell -ExecutionPolicy Bypass -File Plugins/Estuary/ThirdParty/Setup.ps1
    • macOS: bash Plugins/Estuary/ThirdParty/Setup.sh
  2. Rebuild the project. The SDK auto-detects the binaries and uses LiveKit for voice automatically.

When LiveKit is active, the SDK requests a room token, joins the WebRTC room, and routes microphone and TTS audio through it. If the binaries are absent (or Disable LiveKit is set in Project Settings), voice transparently falls back to WebSocket.

Check the current transport at runtime:

EEstuaryVoiceMode Mode = Estuary->GetVoiceMode(); // WebSocket or LiveKit

Echo cancellation settings​

Under Project Settings → Plugins → Estuary → Audio:

SettingDefaultDescription
LiveKit Echo Cancellation (AEC)OnCancels the character's own TTS from the mic path. LiveKit mode only.
AEC Stream Delay (ms)50Latency hint. Raise it if echo persists on high-latency output (e.g., Bluetooth).
Disable LiveKit (WebSocket-only voice)OffForce WebSocket voice; skip all LiveKit token requests and room joins (useful during development).

Mobile Permissions​

On iOS and Android, request microphone access before capturing. Use the latent Acquire Microphone Permission node (under Estuary → Permissions), which fires Granted or Denied:

Event BeginPlay
  → Acquire Microphone Permission
        ├─ Granted ──→ enable your push-to-talk button
        └─ Denied ───→ show a "microphone required" message

On iOS the SDK also configures the audio session for simultaneous record + playback automatically.

Next Steps​

Last updated