Input Components

EstuaryMicrophone

MonoBehaviour -- Captures microphone audio and sends it to the server. Supports both WebSocket (Unity Microphone API) and LiveKit (WebRTC RtcAudioSource) modes.

Namespace: Estuary

Inspector Fields

Field	Type	Default	Description
SampleRate	int	16000	Recording sample rate in Hz
ChunkDurationMs	int	100	Audio chunk size in milliseconds
PushToTalkKey	KeyCode	None	Key for push-to-talk. None = always-on
UseVoiceActivityDetection	bool	false	Enable client-side VAD (WebSocket mode)
VadThreshold	float	0.5	VAD sensitivity: 0 = most sensitive, 1 = least sensitive

Properties

Property	Type	Access	Description
IsRecording	bool	get	true while the microphone is capturing
IsMuted	bool	get	true when muted (LiveKit mode)
CurrentVolume	float	get	Current audio volume level (0--1)
IsSpeechDetected	bool	get	true when VAD detects speech

Methods

Method	Returns	Description
StartRecording()	void	Start capturing audio
StopRecording()	void	Stop capturing audio
Configure(int sampleRate, int chunkDurationMs)	void	Reconfigure audio settings
Mute()	void	Mute the microphone (LiveKit: mutes the track)
Unmute()	void	Unmute the microphone

Events

Event	Signature	Description
OnRecordingStarted	Action	Microphone started capturing
OnRecordingStopped	Action	Microphone stopped capturing
OnVolumeChanged	Action<float>	Volume level changed (0--1)
OnSpeechDetected	Action	User started speaking (VAD)
OnSilenceDetected	Action	User stopped speaking (VAD)

WebSocket vs LiveKit Behavior

Behavior	WebSocket Mode	LiveKit Mode
Capture	Unity Microphone.Start()	LiveKit RtcAudioSource
Transport	Base64 PCM over Socket.IO	WebRTC audio track
AEC	None (manual echo handling)	Native platform AEC
Mute	Stops sending chunks	Mutes the WebRTC track
VAD	Client-side amplitude check	Server-side (Deepgram); client VAD used for interrupt detection

Push-to-Talk

When PushToTalkKey is set to a key other than None:

• WebSocket mode: Audio is only captured and sent while the key is held
• LiveKit mode: The microphone track is muted when the key is released and unmuted when pressed

---

EstuaryWebcam

MonoBehaviour -- Streams camera video to the Estuary world model for spatial awareness and scene understanding.

Namespace: Estuary

Inspector Fields

Field	Type	Default	Description
StreamMode	WebcamStreamMode	LiveKit	LiveKit (WebRTC) or WebSocket (MJPEG)
AutoFallback	bool	true	Fall back to WebSocket if LiveKit is unavailable
TargetFps	int	10	Capture frame rate
TargetWidth	int	1280	Capture width in pixels
TargetHeight	int	720	Capture height in pixels
AutoStartOnConnect	bool	false	Start streaming when the connection is ready
AutoSubscribeSceneGraph	bool	true	Subscribe to scene graph updates automatically
SendPose	bool	false	Send device pose data (AR applications)
CameraTransform	Transform	—	Camera transform for pose data
UseFrontCamera	bool	false	Prefer front-facing camera (mobile/AR)

Properties

Property	Type	Access	Description
IsStreaming	bool	get	true while video is being streamed
WebcamTexture	WebCamTexture	get	The active webcam texture (for preview rendering)
AvailableDevices	WebCamDevice[]	get	List of available camera devices
CurrentSceneGraph	SceneGraph	get	Most recent scene graph

Methods

Method	Returns	Description
StartStreaming(string sessionId)	void	Begin streaming video
StopStreaming()	void	Stop streaming video
SetDevice(string deviceName)	void	Switch to a specific camera by name
SubscribeToSceneGraphAsync()	Task	Subscribe to scene graph updates
UnsubscribeFromSceneGraphAsync()	Task	Unsubscribe from scene graph updates
SendPoseAsync(Matrix4x4 localToWorld)	Task	Send a device pose matrix

Events

Event	Signature	Description
OnSceneGraphUpdated	Action<SceneGraph>	Scene graph update received
OnRoomIdentified	Action<RoomIdentified>	Room identification result received
OnStreamingStarted	Action	Video streaming started
OnStreamingStopped	Action	Video streaming stopped

Streaming Modes

LiveKit mode:

1. Captures frames from WebCamTexture via DirectWebcamVideoSource
2. Publishes a LiveKit video track to the shared room (same room as voice)
3. Notifies the backend to subscribe to the video track (enable_livekit_video)

WebSocket mode:

1. Captures frames from WebCamTexture
2. Encodes frames as JPEG at TargetFps
3. Sends frames as base64-encoded video_frame events over Socket.IO

Scene Graph Updates

When subscribed, the backend processes video frames through its world model pipeline (object detection, scene understanding, spatial reasoning) and returns structured SceneGraph data. See Data Models for the full SceneGraph schema.