Skip to main content

Animation & Lipsync

Drive a MetaHuman face in sync with the character's streaming voice using the server's ARKit-52 blendshape stream.

How It Works​

When animation is enabled, the server streams ARKit-52 facial blendshape frames (bot_animation) alongside the spoken audio, at ~30 fps. The SDK:

  1. Buffers the incoming frames in UEstuaryAnimationReceiver.
  2. Plays them back in sync with the audio clock, so lip movements line up with speech.
  3. Delivers them to your MetaHuman face through a Live Link source — you wire a single Subject Name on the face Animation Blueprint.

The ARKit-52 set is the standard 52 blendshapes used by Apple ARKit and MetaHuman, so it maps cleanly onto a MetaHuman face.

Enabling Animation​

Animation is opt-in and negotiated at connect time.

In Project Settings → Plugins → Estuary → Audio: set Enable Animation to true.

Or per session in code, set the flag on the auth payload before connecting:

FEstuaryAuthPayload Auth;
Auth.CharacterId    = TEXT("your-character-uuid");
Auth.PlayerId       = TEXT("player-123");
Auth.bEnableAnimation = true;     // request the ARKit-52 lipsync stream
Estuary->Connect(ServerUrl, Auth);

Or per character at runtime (before connecting):

CharacterComponent->SetEnableAnimation(true);

:::info Server gate The lipsync stream requires server-side animation to be enabled for your character. After connecting, session_info reports whether the face modality is actually active (AnimationCapabilities.Face). If it's off, the server simply won't send frames — your character still talks, just without lipsync. :::

Wiring the MetaHuman Face​

The SDK delivers blendshapes over Live Link, so setup on the MetaHuman side is minimal:

  1. Make sure the engine's Live Link plugin is enabled (the Estuary plugin enables it as a dependency).
  2. Open your MetaHuman's Face Animation Blueprint (Face_AnimBP).
  3. In the AnimGraph, add or locate a Live Link Pose node feeding the output pose.
  4. Set its Subject Name to the character's Live Link subject. By default this is the character's Character ID; you can override it per character (see below).

That's it — no per-blendshape Property Access bindings, no reparenting. When the character speaks, the Live Link Pose node receives the 52 ARKit weights and animates the face.

Subject naming​

By default the Live Link subject name is the character's Character ID. To disambiguate two characters that share an ID (e.g., two instances of the same agent), set Live Link Subject Override on the UEstuaryCharacterComponent:

CharacterComponent->LiveLinkSubjectOverride = FName(TEXT("NPC_Sarah_Face"));

Resolution order: Override → Character ID → owning actor name.

The Animation Receiver​

UEstuaryAnimationReceiver is created on AEstuaryCharacter automatically and handles all timing. Configurable properties (in the actor's Details panel or via Project Settings defaults):

PropertyDefaultDescription
Remap Asset—Optional DataTable to remap curve names/axes for non-standard face rigs (e.g., MetaHuman Animator imports). Leave empty for standard MetaHuman faces.
First Frame Latency Compensation (ms)150Lead time used to align the first animation frame with the start of audio.
Enable Amplitude Jaw FallbackOnDrives jawOpen from audio loudness when blendshape frames are momentarily absent (utterance start/end, gaps), so the mouth never freezes shut.
Amplitude Jaw Scale1.0Strength of the amplitude-driven jaw.
A2F Crossfade (ms)200Blend window between the amplitude fallback and the full blendshape stream.

The receiver keeps lipsync within ~100 ms of the audio and smoothly settles the face to neutral when a response is interrupted.

Diagnostics​

int32 Drifts = AnimationReceiver->GetDriftExceededCount(); // times drift passed the threshold
float LastMs = AnimationReceiver->GetLastDriftMs();        // most recent audio/anim drift

Body Animation (Experimental)​

:::caution Preview feature Body animation is an experimental preview and is not yet production-ready. The API and content setup are subject to change, and full-body playback requires additional retargeting assets. The sections below describe the intended workflow; for shipping projects, use face lipsync only. :::

In addition to the face, the server can stream full-body pose frames (bot_pose) — a skeletal pose per frame at 30 fps — for gesture and body language synchronized to speech.

  • Enable it with Enable Body Animation in Project Settings, bEnableBodyAnimation on the auth payload, or SetEnableBodyAnimation(true) on the character component.
  • Body animation requires the TTS sample rate to be 16 kHz and server-side body inference enabled for the character; session_info reports AnimationCapabilities.Body so you can check whether it's active.
  • UEstuaryBodyAnimationReceiver consumes the stream and drives a target skeletal mesh. A Finger Policy controls how hand joints are driven (Driven, BindPose, IdleBreath, Statue).

Because body motion is delivered as a generic pose stream, it deliberately trails the voice by a small buffered offset for smooth playback; face lipsync is unaffected.

Troubleshooting​

SymptomLikely cause
Character talks but face doesn't moveAnimation not enabled, or server-side face animation is off for this character (check session_info).
Face moves but lips don't match the rigWrong Subject Name on the Live Link Pose node, or a non-standard rig needs a Remap Asset.
Mouth freezes between phrasesAmplitude jaw fallback disabled — re-enable Enable Amplitude Jaw Fallback.
Face stays frozen after an interruptEnsure you're on the latest SDK; the receiver settles the face to neutral on interrupt automatically.

Next Steps​

Last updated