Camera Module Reference

Complete API reference for camera capture and vision intent detection components.

VisionIntentDetector

Intelligently detects when camera capture should be activated based on user speech.

Location: src/Components/VisionIntentDetector.ts

import { VisionIntentDetector, VisionIntentConfig, VisionIntentResult } from 'estuary-lens-studio-sdk';

Constructor

const detector = new VisionIntentDetector(config: VisionIntentConfig);

VisionIntentConfig

Property	Type	Default	Description
apiKey	string	''	Optional LLM API key (uses heuristic if empty)
debugLogging	boolean	false	Enable debug logging
confidenceThreshold	number	0.7	Threshold for triggering camera (0-1)
endpoint	string	OpenAI URL	Optional LLM API endpoint
model	string	'gpt-4o-mini'	LLM model if using API
customSystemPrompt	string	Built-in prompt	Custom classification prompt

Methods

Method	Parameters	Returns	Description
startListening()	character: EstuaryCharacter	void	Start monitoring character transcripts
stopListening()	-	void	Stop monitoring transcripts
analyzeTranscript()	transcript: string	Promise<VisionIntentResult | null>	Manually analyze text for vision intent
dispose()	-	void	Clean up resources

Properties

Property	Type	Description
isListening	boolean	Whether detector is monitoring transcripts
debugLogging	boolean	Debug mode enabled
confidenceThreshold	number	Current confidence threshold

Events

Event	Payload	Description
visionIntentDetected	(request: CameraCaptureRequest, result: VisionIntentResult)	Vision intent was detected

VisionIntentResult

interface VisionIntentResult {
    requiresVision: boolean;   // Whether visual context is needed
    confidence: number;        // Confidence score (0-1)
    transcript: string;        // Original transcript analyzed
    reason?: string;           // Explanation for the decision
}

Example Usage

import { VisionIntentDetector } from 'estuary-lens-studio-sdk';

const detector = new VisionIntentDetector({
    confidenceThreshold: 0.7,
    debugLogging: true
});

detector.startListening(character);

detector.on('visionIntentDetected', (request, result) => {
    print(`Vision intent detected: ${result.confidence}`);
    print(`Reason: ${result.reason}`);
});

---

VisionIntentDetectorComponent

Lens Studio component wrapper for VisionIntentDetector.

Location: src/Components/VisionIntentDetector.ts

import { VisionIntentDetectorComponent } from 'estuary-lens-studio-sdk';

Inspector Inputs

Input	Type	Default	Description
confidenceThreshold	number	0.7	Threshold for triggering camera (0-1)
debugMode	boolean	true	Enable debug logging
autoConnect	boolean	true	Auto-connect to EstuaryCharacter

Static Properties

Property	Type	Description
instance	VisionIntentDetectorComponent | null	Singleton instance
hasInstance	boolean	Whether instance exists

Properties

Property	Type	Description
detector	VisionIntentDetector | null	Underlying detector instance

Methods

Method	Parameters	Returns	Description
connectToCharacter()	character: EstuaryCharacter	void	Connect to a character
disconnect()	-	void	Disconnect from current character
analyzeTranscript()	transcript: string	void	Manually analyze a transcript
onVisionIntent()	handler: (request, result) => void	() => void	Subscribe to vision intent events (returns unsubscribe)

Example Usage

import { VisionIntentDetectorComponent } from 'estuary-lens-studio-sdk';

@component
export class MyVisionScript extends BaseScriptComponent {
    
    onAwake() {
        const detector = VisionIntentDetectorComponent.instance;
        
        if (detector) {
            const unsubscribe = detector.onVisionIntent((request, result) => {
                print(`Captured request: ${request.request_id}`);
                print(`Confidence: ${result.confidence}`);
            });
        }
    }
}

---

EstuaryVoiceConnection (Example)

Example component that handles voice connection and vision intent detection.

Location: Examples/EstuaryVoiceConnection.ts (class: SimpleAutoConnect)

Example Component

EstuaryVoiceConnection is an example implementation provided in the SDK's Examples/ folder. Copy it to your project and customize as needed.

Inspector Inputs (Vision-Related)

Input	Type	Default	Description
enableVisionIntentDetection	boolean	true	Enable natural language camera activation
visionConfidenceThreshold	number	0.7	Confidence threshold for camera trigger (0-1)

Methods

Method	Parameters	Returns	Description
getVisionIntentDetector()	-	VisionIntentDetector | null	Get the vision intent detector instance

---

EstuaryCamera (Example)

Example component for handling camera capture on Spectacles hardware.

Location: Examples/EstuaryCamera.ts

Example Component

EstuaryCamera is an example implementation provided in the SDK's Examples/ folder. Copy it to your project and customize as needed.

Inspector Inputs

Input	Type	Default	Description
debugMode	boolean	true	Enable debug logging
captureResolution	number	512	Image resolution (smaller dimension in px)
enableVisionAcknowledgment	boolean	true	AI says acknowledgment before analyzing

Methods

Method	Parameters	Returns	Description
manualCapture()	text?: string	void	Manually trigger camera capture

Lifecycle

// CameraModule APIs cannot be called in onAwake!
// Component automatically initializes in OnStartEvent

Example Usage

@component
export class CameraController extends BaseScriptComponent {
    
    @input
    estuaryCamera: ScriptComponent;
    
    onAwake() {
        // Set up a tap to capture
        this.createEvent("TapEvent").bind(() => {
            (this.estuaryCamera as any).manualCapture("What is this?");
        });
    }
}

---

CameraCaptureRequest

Request object sent when camera capture is needed.

Location: src/Core/EstuaryEvents.ts

import { CameraCaptureRequest } from 'estuary-lens-studio-sdk';

Interface

interface CameraCaptureRequest {
    request_id: string;   // Unique identifier for this request
    text?: string;        // Optional text context for the capture
}

Usage

import { EstuaryManager, CameraCaptureRequest } from 'estuary-lens-studio-sdk';

// Receiving camera capture requests
EstuaryManager.instance.on('cameraCaptureRequest', (request: CameraCaptureRequest) => {
    print(`Capture requested: ${request.request_id}`);
    print(`Context: ${request.text}`);
});

---

EstuaryManager Camera Methods

Camera-related methods on the EstuaryManager singleton.

Location: src/Components/EstuaryManager.ts

sendCameraImage

Send a captured image to the server for AI analysis.

sendCameraImage(
    imageBase64: string,
    mimeType?: string,
    requestId?: string,
    text?: string,
    sampleRate?: number
): void

Parameter	Type	Default	Description
imageBase64	string	required	Base64-encoded image data
mimeType	string	'image/jpeg'	MIME type of the image
requestId	string	-	Request ID if responding to server request
text	string	-	Text context to send with image
sampleRate	number	16000	TTS output sample rate

import { EstuaryManager } from 'estuary-lens-studio-sdk';

EstuaryManager.instance.sendCameraImage(
    base64ImageData,
    'image/jpeg',
    'request-123',
    'What do you see?'
);

sendVisionPending

Signal that an image is about to be sent.

sendVisionPending(text: string, requestId?: string): void

Parameter	Type	Description
text	string	Transcript that triggered vision detection
requestId	string	Optional request ID for correlation

import { EstuaryManager } from 'estuary-lens-studio-sdk';

EstuaryManager.instance.sendVisionPending(
    "What do you think of this vase?",
    "vision-intent-12345"
);

updatePreferences

Update session preferences including vision settings.

updatePreferences(preferences: ClientPreferences): void

interface ClientPreferences {
    enableVisionAcknowledgment?: boolean;  // AI acknowledges before analyzing
}

import { EstuaryManager } from 'estuary-lens-studio-sdk';

EstuaryManager.instance.updatePreferences({
    enableVisionAcknowledgment: true
});

---

Events

cameraCaptureRequest

Emitted by EstuaryManager when the server requests a camera capture.

import { EstuaryManager, CameraCaptureRequest } from 'estuary-lens-studio-sdk';

EstuaryManager.instance.on('cameraCaptureRequest', (request: CameraCaptureRequest) => {
    print(`Server requested capture: ${request.request_id}`);
});

visionIntentDetected

Emitted by VisionIntentDetector when vision intent is detected.

import { VisionIntentDetector, CameraCaptureRequest, VisionIntentResult } from 'estuary-lens-studio-sdk';

detector.on('visionIntentDetected', (request: CameraCaptureRequest, result: VisionIntentResult) => {
    print(`Vision intent: ${result.confidence}`);
});

---

Heuristic Detection Keywords

The VisionIntentDetector uses these patterns for heuristic detection:

Strong Indicators (Confidence ~0.9)

const strongIndicators = [
    'look at this', 'look at that', 'see this', 'see that',
    'what is this', 'what is that', 'what\'s this', 'what\'s that',
    'can you see', 'do you see', 'show you',
    'looking at', 'i\'m looking', 'i am looking',
    'this thing', 'that thing', 'what thing',
    'identify this', 'identify that', 'recognize this',
    'what kind of', 'what type of', 'what breed',
    'is this a', 'is that a', 'is this an', 'is that an',
    'read this', 'read that', 'what does this say',
    'help me with this', 'check this out'
];

Medium Indicators (Confidence ~0.5-0.75)

const mediumIndicators = [
    'this', 'that', 'here', 'over here',
    'in front of me', 'right here',
    'what do you think', 'your opinion',
    'is it', 'does it look', 'how does this look'
];

const contextClues = [
    'vase', 'plant', 'flower', 'dog', 'cat', 'animal', 'bird',
    'painting', 'picture', 'photo', 'art', 'artwork',
    'food', 'dish', 'fruit', 'vegetable', 'meal',
    'shirt', 'dress', 'outfit', 'clothes', 'wearing',
    'car', 'building', 'house', 'room', 'landscape',
    'sign', 'text', 'writing', 'label', 'menu',
    'color', 'shape', 'pattern', 'design',
    'ripe', 'fresh', 'broken', 'damaged'
];

When a medium indicator is found with a context clue, confidence is ~0.75. Medium indicator alone gives ~0.5 confidence. Context clue alone gives ~0.4 confidence.

---

Type Definitions

Complete Type Reference

// Vision Intent Configuration
interface VisionIntentConfig {
    apiKey?: string;
    debugLogging?: boolean;
    confidenceThreshold?: number;
    endpoint?: string;
    model?: string;
    customSystemPrompt?: string;
}

// Vision Intent Detection Result
interface VisionIntentResult {
    requiresVision: boolean;
    confidence: number;
    transcript: string;
    reason?: string;
}

// Camera Capture Request
interface CameraCaptureRequest {
    request_id: string;
    text?: string;
}

// Client Preferences
interface ClientPreferences {
    enableVisionAcknowledgment?: boolean;
}

---

SDK Structure

estuary-lens-studio-sdk/
├── src/
│   ├── Components/
│   │   ├── VisionIntentDetector.ts    ← VisionIntentDetector, VisionIntentDetectorComponent
│   │   └── EstuaryManager.ts          ← sendCameraImage, sendVisionPending, updatePreferences
│   └── Core/
│       └── EstuaryEvents.ts           ← CameraCaptureRequest type
└── Examples/
    ├── EstuaryVoiceConnection.ts      ← Vision intent settings (enableVisionIntentDetection, visionConfidenceThreshold)
    └── EstuaryCamera.ts               ← Camera capture settings (captureResolution, enableVisionAcknowledgment)

---

See Also

• Camera Module Guide - Setup and usage guide
• Component Layer - EstuaryManager reference
• Voice Connection - Audio and transcript setup