Action System

Learn how to trigger in-scene actions based on AI character responses using the Estuary SDK's action system.

Overview

The Action System enables your AI characters to trigger real actions in your Lens:

• Play animations when the AI mentions dancing or waving
• Change visuals based on conversation context
• Control game elements through natural conversation
• Trigger UI changes in response to AI actions

┌──────────────────────────────────────────────────────────────┐
│                     Action System Flow                       │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│  AI Response: "Sure, I'll wave at you! <action name="wave"/> │
│                                                              │
│  ┌─────────────────┐    ┌─────────────────────┐              │
│  │ EstuaryCharacter│---→│ EstuaryActionManager│              │
│  │ (Bot Response)  │    │   (Parse Actions)   │              │
│  └─────────────────┘    └────────┬────────────┘              │
│                                  │                           │
│                                  ▼                           │
│                     ┌──────────────────────┐                 │
│                     │   EstuaryActions     │                 │
│                     │  (Global Events)     │                 │
│                     └──────────┬───────────┘                 │
│                                │                             │
│        ┌────────────────────┬──┴────────────────┐            │
│        ▼                    ▼                   ▼            │
│  ┌─────────────┐    ┌─────────────┐    ┌─────────────┐       │
│  │ Script A    │    │ Script B    │    │ Script C    │       │
│  │ onWave()    │    │ onDance()   │    │ onAny()     │       │
│  └─────────────┘    └─────────────┘    └─────────────┘       │
│                                                              │
└──────────────────────────────────────────────────────────────┘

---

Action Tag Format

Actions are embedded in AI responses using XML-like tags:

<action name="wave" />
<action name="sit" />
<action name="dance" />
<action name="jump" />

The SDK automatically parses these tags and dispatches events.

Example AI Response

"I'd be happy to wave at you! <action name="wave" /> There you go!"

The SDK will:

1. Detect the <action name="wave" /> tag
2. Parse the action name (wave)
3. Emit events to all subscribers

---

Quick Start: EstuaryActions Global API

The easiest way to handle actions is using the global EstuaryActions API:

Create an Action Handler Script

import { EstuaryActions, ParsedAction } from 'estuary-lens-studio-sdk';

@component
export class MyActionHandler extends BaseScriptComponent {
    
    private unsubscribes: (() => void)[] = [];
    
    onAwake() {
        // Subscribe to specific actions
        this.unsubscribes.push(
            EstuaryActions.on("wave", (action) => this.onWave(action))
        );
        
        this.unsubscribes.push(
            EstuaryActions.on("dance", (action) => this.onDance(action))
        );
        
        this.unsubscribes.push(
            EstuaryActions.on("sit", (action) => this.onSit(action))
        );
        
        print("[MyActionHandler] Ready for actions!");
    }
    
    onDestroy() {
        // Clean up subscriptions
        this.unsubscribes.forEach(unsub => unsub());
    }
    
    private onWave(action: ParsedAction) {
        print("WAVE action received!");
        // Play wave animation
        this.playAnimation("wave");
    }
    
    private onDance(action: ParsedAction) {
        print("DANCE action received!");
        // Play dance animation
        this.playAnimation("dance");
    }
    
    private onSit(action: ParsedAction) {
        print("SIT action received!");
        // Trigger sit behavior
        this.playAnimation("sit");
    }
    
    private playAnimation(name: string) {
        // Your animation logic here
    }
}

Add to Scene

1. Create a new SceneObject
2. Add your action handler script
3. That's it! No wiring required.

The EstuaryActions system automatically connects when SimpleAutoConnect runs.

---

Subscription Methods

Subscribe to Specific Actions

// Listen for a single action type
const unsubscribe = EstuaryActions.on("wave", (action) => {
    print(`Wave triggered at ${action.timestamp}`);
});

// Later, stop listening
unsubscribe();

Subscribe to All Actions

// Listen for ANY action
const unsubscribe = EstuaryActions.onAny((action) => {
    print(`Action: ${action.name}`);
    
    switch (action.name.toLowerCase()) {
        case "wave":
            this.wave();
            break;
        case "dance":
            this.dance();
            break;
        default:
            print(`Unknown action: ${action.name}`);
    }
});

---

ParsedAction Object

When an action is triggered, you receive a ParsedAction:

interface ParsedAction {
    name: string;       // Action name (e.g., "wave", "sit")
    tag: string;        // Full tag text (e.g., '<action name="wave" />')
    messageId: string;  // ID of the message containing this action
    timestamp: number;  // When the action was detected (Date.now())
}

Example Usage

EstuaryActions.on("jump", (action) => {
    print(`Action: ${action.name}`);
    print(`From message: ${action.messageId}`);
    print(`Detected at: ${new Date(action.timestamp).toISOString()}`);
});

---

Advanced: EstuaryActionManager

For more control, use EstuaryActionManager directly:

Creating a Manager

import { EstuaryActionManager } from 'estuary-lens-studio-sdk';

const actionManager = new EstuaryActionManager(character);
actionManager.debugLogging = true;

Registering Actions (Strict Mode)

In strict mode, only registered actions are triggered:

// Enable strict mode
actionManager.strictMode = true;

// Register allowed actions
actionManager.registerAction("*", "wave", "Wave animation");
actionManager.registerAction("*", "dance", "Dance animation");
actionManager.registerAction("*", "sit", "Sit animation");

// Or register multiple at once
actionManager.registerActions("*", ["jump", "run", "idle"]);

The first parameter is the character ID ("*" means all characters).

Checking Action Registration

if (actionManager.isActionRegistered("*", "wave")) {
    print("Wave action is registered");
}

// Get all registered actions
const actions = actionManager.getRegisteredActions("*");
print(`Registered: ${actions.join(", ")}`);

Enabling/Disabling Actions

// Temporarily disable an action
actionManager.setActionEnabled("*", "dance", false);

// Re-enable it
actionManager.setActionEnabled("*", "dance", true);

---

Action History

The manager keeps a history of triggered actions:

// Get current (most recent) action
const current = actionManager.currentAction;
if (current) {
    print(`Last action: ${current.name}`);
}

// Get full history
const history = actionManager.actionHistory;
print(`Total actions: ${history.length}`);

// Get actions by name
const waves = actionManager.getActionsByName("wave");
print(`Wave count: ${waves.length}`);

// Check if action was triggered recently (within 5 seconds)
if (actionManager.wasActionTriggered("dance", 5000)) {
    print("Dance was triggered in the last 5 seconds");
}

// Clear history
actionManager.clearHistory();

---

Event Types

The action manager emits several events:

actionTriggered

Fired for every action:

actionManager.on('actionTriggered', (action) => {
    print(`Any action: ${action.name}`);
});

action:{name}

Fired for specific action names:

actionManager.on('action:wave', (action) => {
    print("Wave action!");
});

actionManager.on('action:dance', (action) => {
    print("Dance action!");
});

actionsParsed

Fired once per bot response with all actions found:

actionManager.on('actionsParsed', (actions) => {
    print(`Found ${actions.length} action(s) in response`);
    actions.forEach(a => print(`  - ${a.name}`));
});

---

Integration with SimpleAutoConnect

SimpleAutoConnect automatically sets up the action system:

// In SimpleAutoConnect.ts (already done for you):

// Create action manager
this.actionManager = new EstuaryActionManager(this.character);

// Set up global events
EstuaryActions.setManager(this.actionManager);

// Now any script can use EstuaryActions.on()

Accessing the Manager

// If you need direct access:
const autoConnect = /* get reference to SimpleAutoConnect */;
const manager = autoConnect.getActionManager();

---

Practical Examples

Character Animation Controller

import { EstuaryActions, ParsedAction } from 'estuary-lens-studio-sdk';

@component
export class CharacterAnimator extends BaseScriptComponent {
    
    @input
    animationMixer: AnimationMixer;
    
    private currentAnimation: string = "idle";
    private unsubscribe: () => void;
    
    onAwake() {
        this.unsubscribe = EstuaryActions.onAny((action) => {
            this.handleAction(action);
        });
    }
    
    onDestroy() {
        this.unsubscribe?.();
    }
    
    private handleAction(action: ParsedAction) {
        const animName = action.name.toLowerCase();
        
        // Check if we have this animation
        if (this.hasAnimation(animName)) {
            this.playAnimation(animName);
        } else {
            print(`No animation for: ${animName}`);
        }
    }
    
    private hasAnimation(name: string): boolean {
        // Check your animation library
        return ["wave", "dance", "sit", "jump", "idle"].includes(name);
    }
    
    private playAnimation(name: string) {
        if (name === this.currentAnimation) return;
        
        this.currentAnimation = name;
        // Play via AnimationMixer
        // this.animationMixer.setClipWeight(name, 1.0);
        print(`Playing animation: ${name}`);
    }
}

UI State Controller

import { EstuaryActions } from 'estuary-lens-studio-sdk';

@component
export class UIController extends BaseScriptComponent {
    
    @input
    happyIndicator: SceneObject;
    
    @input
    sadIndicator: SceneObject;
    
    @input
    thinkingIndicator: SceneObject;
    
    onAwake() {
        EstuaryActions.on("happy", () => this.showIndicator("happy"));
        EstuaryActions.on("sad", () => this.showIndicator("sad"));
        EstuaryActions.on("thinking", () => this.showIndicator("thinking"));
    }
    
    private showIndicator(type: string) {
        // Hide all
        this.happyIndicator.enabled = false;
        this.sadIndicator.enabled = false;
        this.thinkingIndicator.enabled = false;
        
        // Show selected
        switch (type) {
            case "happy":
                this.happyIndicator.enabled = true;
                break;
            case "sad":
                this.sadIndicator.enabled = true;
                break;
            case "thinking":
                this.thinkingIndicator.enabled = true;
                break;
        }
    }
}

Sound Effects

import { EstuaryActions } from 'estuary-lens-studio-sdk';

@component
export class SoundEffectController extends BaseScriptComponent {
    
    @input
    waveSound: AudioComponent;
    
    @input
    danceMusic: AudioComponent;
    
    onAwake() {
        EstuaryActions.on("wave", () => {
            this.waveSound.play(1);
        });
        
        EstuaryActions.on("dance", () => {
            this.danceMusic.play(1);
        });
        
        EstuaryActions.on("stop_dance", () => {
            this.danceMusic.stop(true);
        });
    }
}

---

Configuring Your AI Character

For actions to work, your AI character needs to be configured to emit action tags. In the Estuary dashboard:

1. Go to your character settings
2. Add action instructions to the system prompt:

When performing physical actions, include action tags in your response.

Available actions:
- <action name="wave" /> - Wave at the user
- <action name="dance" /> - Start dancing
- <action name="sit" /> - Sit down
- <action name="jump" /> - Jump up

Example: "I'd love to dance! <action name="dance" /> Here we go!"

---

Best Practices

Use Meaningful Action Names

// Good: Descriptive names
"wave", "sit_down", "start_dancing", "show_happy_face"

// Bad: Generic names
"action1", "do_thing", "a"

Always Clean Up Subscriptions

private unsubscribes: (() => void)[] = [];

onAwake() {
    this.unsubscribes.push(EstuaryActions.on("wave", this.onWave));
}

onDestroy() {
    this.unsubscribes.forEach(fn => fn());
}

Handle Unknown Actions Gracefully

EstuaryActions.onAny((action) => {
    if (!this.knownActions.includes(action.name)) {
        print(`Unknown action: ${action.name} - ignoring`);
        return;
    }
    this.handleAction(action);
});

Log Actions During Development

// Enable debug logging on the action manager
actionManager.debugLogging = true;

// Or log manually
EstuaryActions.onAny((action) => {
    print(`[ACTION] ${action.name} at ${action.timestamp}`);
});

---

Troubleshooting

Actions Not Triggering

1. Check AI Response: Does it contain <action name="..." />?
2. Check Connection: Is the character connected?
3. Check Subscription: Did you subscribe before the action occurred?
4. Enable Debug Logging: Set debugLogging = true on action manager

Wrong Action Firing

1. Check Action Name: Case sensitivity matters
2. Check Tag Format: Must be <action name="..." />
3. Check Registration: In strict mode, action must be registered

Multiple Subscriptions

If actions fire multiple times:

1. Ensure you're not subscribing multiple times (e.g., in Update loop)
2. Clean up subscriptions in onDestroy()

---

Next Steps

• API Reference - Complete component documentation
• Voice Connection - Audio implementation
• User Management - Conversation persistence