Skip to main content

Voice (WebSocket)

Add real-time voice conversations using the WebSocket voice transport. The SDK captures microphone audio, streams it to the server for speech-to-text, and plays back the character's voice response.

How It Works​

MicrophonePCM → Base64VoiceManagerstream_audioEstuary Serverbot_voiceSpeakerdecode + playAudioPlayer
  1. The SDK requests microphone permission and captures audio at the configured audioSampleRate (default 24 kHz).
  2. Audio is encoded as base64 PCM and streamed to the server via stream_audio WebSocket events.
  3. The server runs speech-to-text and emits sttResponse events with transcription results.
  4. Once a final transcription is produced, the server processes it through the AI pipeline.
  5. The response arrives as botResponse (text) and botVoice (audio) events.
  6. The built-in AudioPlayer decodes and plays the voice audio automatically.

Push-to-talk​

Both WebSocket and LiveKit support explicit press/release turns. Set voiceMode: 'push_to_talk' when constructing the client:

const client = new EstuaryClient({
  serverUrl: 'https://api.estuary-ai.com',
  apiKey: 'est_your_api_key',
  characterId: 'your-character-uuid',
  playerId: 'user-123',
  voiceTransport: 'websocket', // or 'livekit'
  voiceMode: 'push_to_talk',
});

await client.connect();
await client.startVoice();       // Prepare the mic without transmitting
await client.beginPushToTalk();  // Press; await readiness before speaking
// Speak while held. Pauses do not submit your turn.
await client.endPushToTalk();    // Release; finalize the turn
// Repeat begin/end for additional turns.
await client.stopVoice();        // End the call and release the device

Wait for beginPushToTalk() to resolve before showing that speech can begin. Always bind release to pointer-up, pointer-cancel, lost pointer capture, and keyboard key-up/blur as applicable. Release can be called while a press is still pending; it prevents that press from subsequently opening the microphone. Check isPushToTalkActive after awaiting readiness before updating the recording UI. Handle promise rejections from both methods.

startVoice() keeps the microphone acquired between turns. endPushToTalk() flushes the final captured audio block and keeps voice prepared; stopVoice() releases the device. Duplicate press/release calls are safe. Await release before starting the next turn. The character's response uses the ordinary sttResponse, botResponse, and botVoice events.

User mute and suppressMicDuringPlayback remain independent transmission gates. isMuted reports the user mute toggle; isPushToTalkActive reports the held button, including pending readiness. Page resume and reconnect prepare PTT with the button released. To change modes, create a new client and connection: a session that has entered PTT stays in that mode. On a PTT command timeout, reconnect before retrying. Omitting voiceMode uses continuous voice with automatic turn detection.

Starting Voice​

await client.connect();

// Start voice input -- requests microphone permission
await client.startVoice();

By default, the SDK uses voiceTransport: 'auto', which prefers LiveKit if livekit-client is installed, and falls back to WebSocket otherwise. To explicitly use WebSocket:

const client = new EstuaryClient({
  serverUrl: 'https://api.estuary-ai.com',
  apiKey: 'est_your_api_key',
  characterId: 'your-character-uuid',
  playerId: 'user-123',
  voiceTransport: 'websocket',
});
warning

startVoice() requires a browser environment with getUserMedia support. It throws EstuaryError with code MICROPHONE_DENIED if the user denies microphone permission, or VOICE_NOT_SUPPORTED if no voice transport is available.

Speech-to-Text Events​

As the user speaks, the server streams back transcription results:

client.on('sttResponse', (response) => {
  if (response.isFinal) {
    console.log('User said:', response.text);
  } else {
    // Interim transcription -- useful for showing live captions
    console.log('Hearing:', response.text);
  }
});

After a final sttResponse, the server automatically triggers the AI pipeline. The response flows through the same botResponse and botVoice events used by text chat.

Bot Voice Audio​

Voice audio arrives as base64-encoded PCM chunks via the botVoice event:

client.on('botVoice', (voice) => {
  console.log(`Audio chunk ${voice.chunkIndex} for message ${voice.messageId}`);
  if (voice.isFinal) {
    console.log('Last audio chunk received');
  }
});

In a browser environment, the SDK's built-in AudioPlayer automatically decodes and plays these chunks. You do not need to handle playback manually unless you want custom audio processing.

Playback Events​

Track audio playback state with these events:

client.on('audioPlaybackStarted', (messageId) => {
  console.log('Started playing audio for:', messageId);
});

client.on('audioPlaybackComplete', (messageId) => {
  console.log('Finished playing audio for:', messageId);
  // The SDK automatically notifies the server via audio_playback_complete
});

The audioPlaybackComplete notification tells the server that the client has finished playing the audio, which allows the server to track conversation pacing.

Stopping Voice​

await client.stopVoice();

This stops the microphone, cleans up the voice manager, and emits a voiceStopped event. The client remains connected for text chat. stopVoice() returns a Promise<void> -- await it if you need to know the voice manager has fully shut down before continuing.

Muting​

Toggle the microphone without stopping the voice session:

client.toggleMute();
console.log('Muted:', client.isMuted);

When muted, the microphone stream is paused but the voice session remains active. The character's voice responses continue to play.

info

toggleMute() throws EstuaryError with code VOICE_NOT_ACTIVE if voice has not been started.

Voice Lifecycle Events​

client.on('voiceStarted', () => {
  console.log('Voice session started');
  showMicrophoneIndicator();
});

client.on('voiceStopped', () => {
  console.log('Voice session stopped');
  hideMicrophoneIndicator();
});

Example: Voice Chat​

import { EstuaryClient } from '@estuary-ai/sdk';

const client = new EstuaryClient({
  serverUrl: 'https://api.estuary-ai.com',
  apiKey: 'est_your_api_key',
  characterId: 'your-character-uuid',
  playerId: 'user-123',
  voiceTransport: 'websocket',
});

client.on('sttResponse', (response) => {
  if (response.isFinal) {
    console.log('You:', response.text);
  }
});

client.on('botResponse', (response) => {
  if (response.isFinal) {
    console.log('Bot:', response.text);
  }
});

client.on('error', (err) => {
  console.error('Error:', err.message);
});

async function main() {
  await client.connect();
  console.log('Connected! Starting voice...');

  await client.startVoice();
  console.log('Listening -- speak into your microphone.');
}

main().catch(console.error);

Next Steps​

Last updated