Vision & Camera Protocol

Estuary characters can see the user's environment through a camera feed. This page documents the Socket.IO events for camera image capture and vision configuration.

All events use the /sdk namespace. See Conversation Protocol for connection setup.

---

Camera Image Submission

When the SDK has a camera image to send (either proactively or in response to a server request), it emits a camera_image event.

Client → Server

Event: camera_image
Payload: {
  "image": "<base64-encoded image>",
  "mime_type": "image/jpeg",
  "request_id": "req_abc123",
  "text": "What do you see?",
  "sample_rate": 24000
}

Field	Type	Required	Description
image	string	Yes	Base64-encoded image data
mime_type	string	Yes	MIME type of the image, e.g. "image/jpeg" or "image/png"
request_id	string	No	Matches the request_id from a camera_capture request (if this image was server-initiated)
text	string	No	Optional user question about the image
sample_rate	number	No	Preferred TTS sample rate (Hz) for the voice reply. Falls back to the session's audio_sample_rate if omitted.

The server analyzes the image using the character's configured VLM and streams a bot_response with the character's reaction.

---

Server-Initiated Capture

The server can request the client to capture and send a camera image. This is triggered by the character's agentic tool system -- when the LLM decides it needs to see the user's environment, it invokes a camera capture tool.

Server → Client

Event: camera_capture
Payload: {
  "request_id": "req_abc123",
  "text": "Can you show me what's on your desk?"
}

Field	Type	Description
request_id	string	Unique ID for this capture request -- include in the camera_image response
text	string | null	The user's original question that prompted the capture

Expected Flow

1. Server emits camera_capture to the client
2. Client captures an image from the device camera
3. Client emits camera_image with the captured image and the request_id
4. Server analyzes the image and streams a bot_response

---

Vision Acknowledgement

By default, when the character requests a camera image (via camera_capture), it speaks a short verbal acknowledgement before the image is captured and analyzed (e.g., "let me take a look"). The update_preferences event lets you toggle this behaviour at runtime.

Client → Server

Event: update_preferences
Payload: { "enableVisionAcknowledgment": true }

Field	Type	Description
enableVisionAcknowledgment	boolean	When true, the character speaks an acknowledgement before analyzing a camera image. When false, the analysis happens silently and only the resulting reply is spoken.

This preference applies to the current session only. Reissue it after reconnecting if you want to keep the override.

---

Next Steps

• Conversation Protocol -- Text and voice event reference
• Action Protocol -- Actions triggered from vision context
• Memory System -- How visual context is remembered