Vision

Let an Estuary character "see" — send an image from the rendered scene or a webcam and get a spoken description back.

How It Works

UEstuaryCameraComponent captures a frame, encodes it as JPEG, and sends it to the server. The character's vision-language model analyzes it and the result streams back as a normal response. Capture works two ways:

• On demand — you call Send Camera Frame from gameplay (e.g., the player presses "show").
• Server-initiated — the server detects visual intent in the conversation ("what am I holding?") and asks the client to capture. The component captures automatically if the user has granted consent.

Two capture sources are supported: the rendered scene (what an in-world camera sees) and a webcam (the player's real camera).

---

Setup

1. On your character actor, Add Component → Estuary Camera Component.
2. Position it where you want the in-world "eye" to look (for rendered-scene capture), or configure it for webcam capture (below).
3. Grant capture consent before sending any frames:

CameraComponent->SetCameraConsent(true);

:::caution Consent is required No frame is sent — including server-initiated captures — unless consent has been granted via SetCameraConsent(true). This is a privacy gate; surface it in your UI. :::

---

Capturing the Rendered Scene

By default the component captures the rendered scene through an internal scene-capture and render target.

// Optionally set the capture resolution
CameraComponent->SetRenderTargetSize(1280, 720);

// Send the current view to the character
bool bSent = CameraComponent->SendCameraFrame();

In Blueprint: call Send Camera Frame on the component. The frame is encoded off the game thread and emitted to the server.

---

Capturing a Webcam

Switch the capture source to the player's webcam:

CameraComponent->SetCaptureSource(EEstuaryCaptureSource::Webcam);
CameraComponent->StartWebcam(/*DeviceIndex=*/0);

// ...later, send a webcam frame
CameraComponent->SendWebcamFrame();

// When finished
CameraComponent->StopWebcam();

You can flip the active source at any time with SetCaptureSource, or toggle all registered cameras from the subsystem:

EEstuaryCaptureSource NewSource = Estuary->ToggleCaptureSource();
FText Label = Estuary->GetCaptureSourceLabel(); // e.g. "Camera: Webcam"

---

Receiving the Vision Result

Bind On Vision Response on the UEstuaryCharacterComponent. The result streams in chunks like a normal response; Description accumulates the text and bIsFinal marks the end.

CharacterComponent->OnVisionResponse.AddDynamic(this, &AMyActor::HandleVision);

void AMyActor::HandleVision(const FEstuaryVisionResponse& Response)
{
    AccumulatedDescription += Response.Description;
    if (Response.bIsFinal)
    {
        UE_LOG(LogTemp, Log, TEXT("Character sees: %s"), *AccumulatedDescription);
        AccumulatedDescription.Reset();
    }
}

The character may also speak the description aloud if vision acknowledgment is enabled (see Preferences).

---

Server-Initiated Capture

When the conversation implies the character should look (e.g., the user says "look at this"), the server sends a capture request. Bind On Camera Capture Requested to observe it:

CharacterComponent->OnCameraCaptureRequested.AddDynamic(this, &AMyActor::HandleCaptureRequest);

void AMyActor::HandleCaptureRequest(const FEstuaryCameraCapture& Capture)
{
    // The camera component already auto-captured if consent was granted.
    // Bind here to react, or to prompt the user if consent is missing.
}

If consent has been granted, the camera component captures and sends automatically — you don't need to call SendCameraFrame yourself. Bind this event to handle the no-consent case (e.g., show a permission prompt).

---

Configuration

Setting	Where	Default	Description
JPEG Quality	Component / Project Settings	75	1–100. Higher = better recognition but larger payload.
Capture Source	Component	Rendered Scene	RenderedScene or Webcam.
Has User Consent	Component (read-only)	false	Set via SetCameraConsent.

On mobile, request camera permission first with the latent Acquire Camera Permission node (under Estuary → Permissions), which fires Granted or Denied.

---

Next Steps

• Text Chat — Preferences and vision acknowledgment
• API Reference: Camera Component
• API Reference: Data Models