Skip to main content

Voice Connection

The Estuary Unity SDK supports two voice transports: LiveKit (WebRTC) and WebSocket. LiveKit is recommended for production use because it provides lower latency and native acoustic echo cancellation (AEC).

Voice Modes​

ModeTransportLatencyAECSetup
LiveKitWebRTCLowNative platform AECRequires LiveKit server
WebSocketSocket.IOHigherManual/noneWorks everywhere

Set the mode in your EstuaryConfig asset or at runtime:

EstuaryManager.Instance.Config.VoiceMode = VoiceMode.LiveKit;

WebSocket Voice​

How It Works​

  1. Client captures audio via Unity's Microphone API
  2. Audio is converted to PCM 16-bit, base64-encoded, and sent as stream_audio events
  3. Server runs speech-to-text on the audio stream
  4. Server sends back stt_response events with transcription
  5. On final transcription, server generates a response and streams bot_response + bot_voice events
  6. Client decodes base64 audio and plays via AudioSource

Setup​

Add EstuaryMicrophone and EstuaryAudioSource components to your character. The microphone captures at 16 kHz in 100 ms chunks (the rate required by the backend STT pipeline) and uses Unity's AudioSettings.outputSampleRate for TTS playback (24 kHz default per the SDK contract).

Starting a Voice Session​

// Automatically (on the EstuaryCharacter component)
autoStartVoiceSession = true;

// Or manually
character.StartVoiceSession();

StartVoiceSession() sends start_voice to the server (enabling server-side STT) and starts the microphone. When you call EndVoiceSession(), it sends stop_voice and stops the mic.

STT Responses​

character.OnTranscript += (SttResponse stt) =>
{
    if (stt.IsFinal)
    {
        Debug.Log($"User said: {stt.Text}");
    }
    else
    {
        // Interim result (real-time as user speaks)
        transcriptLabel.text = stt.Text;
    }
};

LiveKit Voice​

How It Works​

  1. After Socket.IO session is established, SDK requests a LiveKit token
  2. Server returns a JWT token, LiveKit URL, and room name
  3. SDK connects to the LiveKit room using the LiveKit Unity SDK
  4. SDK notifies the server it has joined (livekit_join)
  5. Server joins the bot to the same room (livekit_ready)
  6. Audio flows bidirectionally over WebRTC:
    • Client publishes a microphone track (with native AEC)
    • Client receives the bot's TTS audio track

Setup​

LiveKit mode uses the same EstuaryMicrophone and EstuaryAudioSource components, but the audio transport is different:

  • Microphone: Uses native WebRTC capture through RtcAudioSource instead of Unity's Microphone API. This enables proper AEC.
  • Playback: LiveKit's AudioStream handles playback automatically through an AudioSource it creates at runtime.

Connection Flow​

// Auto-connect flow (recommended):
// 1. EstuaryCharacter.autoConnect = true
// 2. EstuaryCharacter.autoStartVoiceSession = true
// 3. EstuaryConfig.voiceMode = VoiceMode.LiveKit
//
// The SDK handles the full flow automatically:
// Connect → session_info → request token → join room → livekit_ready → publish mic

// Manual flow:
character.OnConnected += async (session) =>
{
    // Request LiveKit token
    await EstuaryManager.Instance.RequestLiveKitTokenAsync();
};

EstuaryManager.Instance.OnLiveKitReady += (roomName) =>
{
    Debug.Log($"LiveKit ready in room: {roomName}");
    // Audio is now flowing
};

LiveKit Connection States​

Monitor the LiveKit connection via EstuaryManager:

EstuaryManager.Instance.OnLiveKitStateChanged += (state) =>
{
    // LiveKitConnectionState:
    // Disconnected → RequestingToken → Connecting →
    // WaitingForBot → Ready
    statusLabel.text = state.ToString();
};

EstuaryAudioSource​

The EstuaryAudioSource component handles TTS playback with a ring buffer for smooth streaming.

WebSocket Mode​

In WebSocket mode, audio arrives as base64-encoded PCM chunks in bot_voice events. The component:

  1. Decodes base64 to PCM16 bytes
  2. Converts to float samples
  3. Resamples if the server sample rate differs from Unity's output rate
  4. Writes to a ring buffer
  5. Plays via a streaming AudioClip with a PCM reader callback

The ring buffer provides gap-free playback even when network jitter causes irregular chunk delivery.

LiveKit Mode​

In LiveKit mode, EstuaryAudioSource defers to LiveKit's AudioStream, which handles playback automatically. The component tracks message_id for interrupt handling.

Interrupts​

When the user speaks while the bot is talking, the server confirms the barge-in and sends an interrupt event. The SDK then stops playback and clears the audio buffer. Use character.OnInterrupt to react to the confirmed interruption.

Events​

audioSource.OnPlaybackStarted += () => { lipSync.Enable(); };
audioSource.OnPlaybackComplete += () => { lipSync.Disable(); };
audioSource.OnPlaybackInterrupted += () => { lipSync.Disable(); };

EstuaryMicrophone​

Speech detection​

Speech detection and turn completion run on the server in both voice modes. The microphone sends audio continuously during a voice session unless it is muted or push-to-talk is active. You do not need to configure a local VAD threshold.

Push-to-Talk​

Enable PushToTalkEnabled on EstuaryMicrophone, then optionally set PushToTalkKey (a KeyCode). With no key, call PushToTalkPress() / PushToTalkRelease() from your own input (touch/XR buttons). The key field is disabled in the Inspector while PTT is off, and an assigned key is ignored in that mode. Existing scenes that relied on a key alone must also enable the checkbox. Push-to-talk is server-side (contract v1.11): the press interrupts any in-flight bot speech and signals the server to hold end-of-turn dispatch while the button is down; the release makes the server finalize the transcript and dispatch exactly one turn — a pause mid-hold no longer makes the character answer over the held button. The talk button only works during an active voice session (StartVoiceSession()); presses outside one are ignored.

  • LiveKit mode: press unmutes the WebRTC track, release mutes it. The mic starts muted until the first press.
  • WebSocket mode: audio chunks are only sent while the button is held.

Events​

microphone.OnRecordingStarted += () => { micIcon.SetActive(true); };
microphone.OnRecordingStopped += () => { micIcon.SetActive(false); };
// Volume monitoring is available in WebSocket mode.
microphone.OnVolumeChanged += (float vol) => { volumeMeter.value = vol; };

Lip Sync Integration​

In LiveKit mode, the bot's AudioSource is created at runtime when the bot joins the room. To integrate lip sync:

EstuaryManager.Instance.OnBotAudioSourceCreated += (AudioSource source) =>
{
    // Attach your lip sync analyzer to this AudioSource
    lipSyncAnalyzer.SetAudioSource(source);
};

Next Steps​

Last updated