Voice Connection

Learn how to implement real-time voice conversations with AI characters using the Estuary SDK.

Overview

The Estuary SDK provides full duplex voice communication:

• Voice Input: Capture user speech via microphone → Speech-to-Text
• Voice Output: AI responses → Text-to-Speech → Audio playback

---

Quick Start with EstuaryVoiceConnection

The easiest way to implement voice is using the EstuaryVoiceConnection example component shipped in the SDK's Examples/ folder:

Step 1: Scene Setup

Create these SceneObjects:

Object Name	Components
Estuary Credentials	EstuaryCredentials script
Estuary Connection	EstuaryVoiceConnection script
Microphone	MicrophoneRecorder script
Audio Output	DynamicAudioOutput script + AudioComponent
Internet Module	InternetModule resource

Step 2: Configure EstuaryVoiceConnection

In the Inspector, connect:

credentialsObject        → Estuary Credentials
internetModule           → Internet Module
microphoneRecorderObject → Microphone
dynamicAudioOutputObject → Audio Output

That's It!

EstuaryVoiceConnection handles:

• Connection management
• Microphone streaming
• Voice Activity Detection (server-side)
• Audio playback
• Interrupt handling
• Auto-reconnection

---

Manual Voice Implementation

For more control, you can implement voice manually.

Set Up the Character

import { EstuaryCharacter } from 'estuary-lens-studio-sdk';
import { EstuaryConfig } from 'estuary-lens-studio-sdk';
import { EstuaryManager } from 'estuary-lens-studio-sdk';

@component
export class VoiceController extends BaseScriptComponent {
    
    @input
    internetModule: InternetModule;
    
    private character: EstuaryCharacter;
    
    onAwake() {
        // REQUIRED: Set the InternetModule used for WebSocket connections
        // (Lens Studio 5.9+ exposes createWebSocket on InternetModule)
        EstuaryManager.instance.internetModule = this.internetModule;
        
        // Create character
        this.character = new EstuaryCharacter(
            "your-character-id",
            "unique-player-id"
        );
        
        // Configure and connect
        const config: EstuaryConfig = {
            serverUrl: "wss://api.estuary-ai.com",
            apiKey: "your-api-key",
            characterId: "your-character-id",
            playerId: "unique-player-id",
            debugLogging: true
        };
        
        this.character.initialize(config);
    }
}

Set Up Microphone Input

import { EstuaryMicrophone, MicrophoneRecorder } from 'estuary-lens-studio-sdk';

@component
export class VoiceController extends BaseScriptComponent {
    
    @input
    microphoneRecorderObject: SceneObject;
    
    private microphone: EstuaryMicrophone;
    
    onAwake() {
        // ... character setup ...
        
        // Create microphone handler
        this.microphone = new EstuaryMicrophone(this.character);
        this.microphone.debugLogging = true;
        
        // Find MicrophoneRecorder on the SceneObject
        const recorder = this.findMicrophoneRecorder(this.microphoneRecorderObject);
        if (recorder) {
            this.microphone.setMicrophoneRecorder(recorder);
        }
        
        // Connect microphone to character
        this.character.microphone = this.microphone;
    }
    
    private findMicrophoneRecorder(obj: SceneObject): MicrophoneRecorder | null {
        const scripts = obj.getComponents("Component.ScriptComponent") as any[];
        for (let i = 0; i < scripts.length; i++) {
            const comp = scripts[i] as any;
            if (comp?.onAudioFrame && typeof comp.startRecording === 'function') {
                return comp as MicrophoneRecorder;
            }
        }
        return null;
    }
}

Handle Voice Events

private setupEventHandlers() {
    // When connected, start voice session
    this.character.on('connected', (session) => {
        print(`Connected: ${session.sessionId}`);
        
        // IMPORTANT: Start voice session before streaming audio
        this.character.startVoiceSession();
        
        // Start microphone
        this.microphone.startRecording();
    });
    
    // Handle transcription (what user said)
    this.character.on('transcript', (stt) => {
        if (stt.isFinal) {
            print(`[You] ${stt.text}`);
        }
    });
    
    // Handle AI text response
    this.character.on('botResponse', (response) => {
        if (response.isFinal) {
            print(`[AI] ${response.text}`);
        }
    });
    
    // Handle AI voice response
    this.character.on('voiceReceived', (voice) => {
        // Play audio via DynamicAudioOutput
        this.playVoiceAudio(voice);
    });
    
    // Handle interrupts (user speaks while AI is talking)
    this.character.on('interrupt', (data: InterruptData) => {
        // Stop current audio playback
        this.stopAudioPlayback();
        print(`Interrupted message ${data.messageId}: ${data.reason}`);
    });
}

Audio Playback

interface DynamicAudioOutput {
    initialize(sampleRate: number): void;
    addAudioFrame(uint8Array: Uint8Array, channels: number): void;
    interruptAudioOutput(): void;
}

private dynamicAudioOutput: DynamicAudioOutput;
private audioInitialized: boolean = false;

private setupAudioOutput(audioOutputObject: SceneObject) {
    // Find DynamicAudioOutput component
    const scripts = audioOutputObject.getComponents("Component.ScriptComponent") as any[];
    for (let i = 0; i < scripts.length; i++) {
        const comp = scripts[i] as any;
        if (comp?.initialize && comp?.addAudioFrame) {
            this.dynamicAudioOutput = comp;
            break;
        }
    }

    if (this.dynamicAudioOutput) {
        // Initialize with TTS sample rate (24kHz)
        this.dynamicAudioOutput.initialize(24000);
        this.audioInitialized = true;

        // Required for low-latency TTS playback on Spectacles.
        // LowLatency playbackMode is silently downgraded to LowPower if any
        // AudioComponent in the scene has mixToSnap=false — set it BEFORE
        // playbackMode so the change is coherent.
        const audioComp = audioOutputObject.getComponent("Component.AudioComponent") as AudioComponent;
        if (audioComp) {
            (audioComp as any).mixToSnap = true;
            audioComp.playbackMode = Audio.PlaybackMode.LowLatency;
        }
    }
}

private playVoiceAudio(voice: BotVoice) {
    if (!this.dynamicAudioOutput || !voice.audio) return;
    
    // Decode Base64 to PCM bytes
    const pcmBytes = Base64.decode(voice.audio);
    
    // Play audio (mono = 1 channel)
    this.dynamicAudioOutput.addAudioFrame(pcmBytes, 1);
}

private stopAudioPlayback() {
    if (this.dynamicAudioOutput) {
        this.dynamicAudioOutput.interruptAudioOutput();
    }
}

---

Voice Session Management

Starting a Voice Session

You must start a voice session before streaming audio:

character.on('connected', (session) => {
    // This enables audio streaming to the server
    character.startVoiceSession();
    
    // Now you can start recording
    microphone.startRecording();
});

Ending a Voice Session

// Stop voice input (but keep connection open)
character.endVoiceSession();
microphone.stopRecording();

Voice Session State

// Check if voice session is active
if (character.isVoiceSessionActive) {
    // Audio streaming is enabled
}

---

Handling Interrupts

When the user speaks while the AI is responding, an interrupt is triggered:

character.on('interrupt', (data: InterruptData) => {
    // 1. Stop audio playback immediately
    dynamicAudioOutput.interruptAudioOutput();
    
    // 2. Clear any pending response text
    // 3. Update UI to show user is speaking
    
    print(`Interrupted message ${data.messageId}: ${data.reason}`);
});

The server automatically:

• Stops generating the current response
• Clears the TTS queue
• Starts processing the new user input

---

WebSocket Send Queue

The Estuary SDK enforces a 75 ms minimum gap between WebSocket sends to prevent protocol corruption on Spectacles hardware. This is managed internally via a send queue.

You must call EstuaryManager.instance.tick() periodically (e.g., from an UpdateEvent) to process this queue. Without this, ping/pong responses may not be sent during silence, causing connection timeouts.

// In your main script
this.createEvent("UpdateEvent").bind(() => {
    EstuaryManager.instance.tick();
});

note

The EstuaryVoiceConnection example component already calls tick() automatically. You only need to add this if implementing a custom connection flow.

---

Text-Only Fallback

You can also send text messages without voice:

// Send text directly to the AI
character.sendText("Hello, how are you?");

// Listen for text response
character.on('botResponse', (response) => {
    if (response.isFinal) {
        print(`AI responded: ${response.text}`);
    }
});

---

Audio Format Details

Input Audio (Microphone to Server)

Property	Value
Sample Rate	16,000 Hz
Format	16-bit PCM (signed, little-endian)
Channels	Mono (1)
Encoding	Base64 string
Chunk Size	~100ms of audio

Output Audio (Server to Speaker)

Property	Value
Sample Rate	24,000 Hz
Format	16-bit PCM (signed, little-endian)
Channels	Mono (1)
Encoding	Base64 string

---

Troubleshooting

No Audio Being Sent

WARNING: Audio dropped: voice session not active! Call startVoiceSession() first.

Solution: Call character.startVoiceSession() after connection before recording.

No Audio Playback

1. Ensure DynamicAudioOutput has an AudioComponent attached
2. Verify an AudioTrack asset is assigned
3. Check initialize() was called with correct sample rate (24000)

Audio Cuts Out

The SDK includes throttling to prevent WebSocket buffer overflow. If audio still cuts out:

• Check network stability
• Reduce other network traffic in your Lens

WebSocket Unavailable Error

WebSocket not available in this Lens Studio Preview build.

WebSocket and HTTP networking are supported in Lens Studio Preview as of Lens Studio 5.12+, so you can test conversations without flashing to hardware. If you see this error, your installed Lens Studio is older than 5.12 — upgrade Lens Studio, or deploy the Lens to Spectacles to test.

---

Best Practices

Always Initialize Audio Early

// Initialize audio output immediately after finding the component
dynamicAudioOutput.initialize(24000);

Handle Connection Loss

character.on('disconnected', () => {
    microphone.stopRecording();
    // Show UI indicator
});

character.on('error', (error) => {
    print(`Voice error: ${error}`);
    // Attempt recovery or show message
});

Provide Visual Feedback

Users benefit from knowing:

• When their voice is being captured
• When the AI is "thinking"
• When the AI is speaking

Respect Voice Sessions

// Don't stream audio without an active session
if (character.isConnected && character.isVoiceSessionActive) {
    character.streamAudio(audioBase64);
}

---

Next Steps

• User Management - Persist conversations across sessions
• Action System - Trigger actions from voice responses
• API Reference - Detailed component documentation