README_lipsync.md

Ppaso-TTS Lipsync — companion animation module

한국어: README_lipsync_ko.md · Main: README.md

Blendshape mapping detail: README_bs2d.md

Pure-numpy companion module ppaso_lipsync (in example/) turns Ppaso-TTS output into per-frame 2D mouth + 1D eye-blink animation curves. A single lipsync curve drives two render routes:

• atlas — 2D sprite swap (nearest viseme → one mouth sprite). 07_avatar_video.py
• blendshape — 3D morph-target blend via bs2d. 08_vrm_video.py

Demo

Blendshape — 3D VRM, lipsync + body idle (samples/08_vrm_lipsync.mp4, download):

Atlas — 2D sprite (samples/06_lipsync_intro.mp4, download):

엣지디바이스용 경량화 립싱크 애니메이션 기능. 동일한 커브가 2D 스프라이트와 3D 블렌드쉐입 애니메이션을 모두 구동합니다. (One curve drives both 2D sprite and 3D blendshape animation.)

How it works

TTS phones + durations
        │
        ▼
  viseme decompose   ← 9 anchor (A/I/U/E/O/EU/EO/M/sil) + j/w glide,
        │              palatalized / alveolar / sibilant / velar / glottal
        ▼
  keyframes → 2D xy curve     ← shared lipsync curve, x∈[-1,1] y∈[-0.1,1]
        │
        ├──────────►  atlas route      : nearest viseme → sprite swap
        │
        └──────────►  blendshape route : bs2d → barycentric → morph blend

  Eye blink (independent)  → state_timeline {0:open,1:half,2:close}
                             + curve_timeline [0,1]

The lipsync curve generation (decompose → keyframes → interpolate) is shared. The two routes differ only in how they consume the (x, y) curve — see the atlas/blendshape distinction below.

Route A — atlas (2D sprite)

Each frame the curve point picks the nearest viseme and the renderer shows that one mouth sprite (sprites cannot blend). Best for 2D character art with a sprite sheet.

cd example
python 07_avatar_video.py --text "안녕하세요." --out hi.mp4

• Sample 2D avatar: example/avatar/ (atlas.json + sprite sheet).
• Curve → sprite mapping: atlas.json viseme_to_sprite.

Route B — blendshape (3D VRM)

For 3D characters with morph-target (blendshape) visemes. Here the curve goes through bs2d, which finds the triangle containing the point and returns barycentric blend weights — so 3 blendshapes are mixed per frame, giving continuous mouth motion (a sprite swap cannot do this).

lipsync (x,y) curve → bs2d(target="vrm") → {a,i,u,e,o,neutral} weight
                    → VRM blendShapeMaster morph → mesh deform → render

• vrm-spec — bs2d maps the curve onto a standard VRM 0.x avatar's 5 viseme + neutral expressions (Korean ㅏㅣㅜㅔㅗ → VRM a/i/u/e/o).
• Three render backends (--backend):
  • gles (default) — EGL + OpenGL ES 3.0. Real GPU acceleration on ARM SoCs (RK3576 Mali-G52 etc.). Needs PyOpenGL.
  • egl — moderngl + EGL desktop GL 3.3. For x86 GPUs / Mesa llvmpipe; ARM GPU drivers usually cap desktop GL at 3.1 so moderngl can't run there — use gles on-device. Needs moderngl + glcontext.
  • numpy — pure-numpy software rasterizer, zero GPU/driver dependency (slow — fallback only).
• Bundled sample VRM: example/avatar_vrm/AvatarSample_F.vrm (VRoid 'Vita', CC0).

cd example
python 08_vrm_video.py --text "안녕하세요." --out hi.mp4   # gles (default)
python 08_vrm_video.py --backend egl             # x86 desktop GPU
python 08_vrm_video.py --backend numpy           # no-GPU fallback

bs2d full-spec ↔ vrm-spec, the projection / barycentric stages, and how to map your own character's blendshapes (9 or more) — see README_bs2d.md.

Body animation (optional)

09_vrm_live.py and 10_vrm_live_streaming.py add a skeletal body idle on top of lipsync — GPU skinning via raster_gles_skinned. It needs a body clip (--idle clip.npz); without one the body stays at rest pose (lipsync and video still render fine).

Make a clip from any FBX skeletal animation with tools/fbx_to_clip.py (needs bpy — pip install bpy):

python tools/fbx_to_clip.py  idle.fbx  avatar_vrm/idle.npz
python 09_vrm_live.py --idle avatar_vrm/idle.npz

The clip is not bundled — animation assets such as Mixamo are royalty-free to embed in your rendered output, but their raw motion data is generally not redistributable, so supply your own. BodyClipPlayer assumes a loop-authored clip; fbx_to_clip.py drops the duplicate loop endpoint.

Quick start

from ppaso_tts import PpasoTTS
from ppaso_lipsync import generate_blink_timeline

tts = PpasoTTS('./', backend='onnx')          # or 'rknn'
wav, lipsync = tts.synthesize_with_lipsync('안녕하세요.', fps=30)
# lipsync.xy_timeline   → (n_frames, 2)  — feeds atlas or blendshape route
# lipsync.keyframes     → viseme keyframes (debug)

blink = generate_blink_timeline(duration=lipsync.duration, fps=30, seed=42)
# blink.state_timeline / blink.curve_timeline

Streaming (chunk-to-chunk continuity is automatic):

from ppaso_tts import PpasoTTS, StreamingTTS

stream = StreamingTTS(PpasoTTS('./', backend='rknn'))
for token in llm_token_stream():
    for wav, lipsync in stream.feed_with_lipsync(token):   # emits on .!?
        play_audio(wav); animate(lipsync)
for wav, lipsync in stream.flush_with_lipsync():
    ...

Performance

Atlas route — RK3576 measured

End-to-end (TTS + lipsync + eye blink + sprite composite at 768×1152, no ffmpeg encode). RK3576, best-of-5 @ 30 fps.

Case	Input	wav	Backend	E2E	RTF	Speed
short	21 chars	3.49 s	RK3576 NPU	864 ms	0.247	4.0× RT
			ONNX CPU (4t)	897 ms	0.256	3.9× RT
medium	41 chars	7.26 s	RK3576 NPU	1867 ms	0.257	3.9× RT
long	110 chars	16.65 s	RK3576 NPU	3668 ms	0.220	4.5× RT

Reproduce: cd example && python bench_lipsync.py --backend rknn --model-dir ... Lipsync curve generation itself adds < 1 ms on top of TTS.

Blendshape route — RK3576 measured

End-to-end (TTS + lipsync + eye blink + morph + render at 512×768, no ffmpeg encode — same E2E definition as the atlas route above). RK3576 Mali-G52, gles backend, best-of-3 @ 30 fps. Two modes: lipsync only (static skeleton — 08_vrm_video.py) and lipsync + body (GPU skinning + body idle animation — 09_vrm_live.py).

① lipsync only — E2E by text length:

Case	wav	frames	E2E	RTF
short	3.51 s	110	2222 ms	0.63
medium	7.26 s	223	4637 ms	0.64
long	16.31 s	497	9358 ms	0.57

② lipsync + body — E2E by text length:

Case	wav	frames	E2E	RTF
short	3.51 s	110	2809 ms	0.80
medium	7.26 s	223	5580 ms	0.77
long	16.31 s	497	12314 ms	0.76

③ lipsync + body — medium (7.26 s, 223 frames) stage breakdown:

Stage	medium
TTS + lipsync	1100 ms
Eye blink	0.5 ms
FK + pose (skeleton)	1031 ms (4.6 ms/frame)
Morph (bs2d weight)	19 ms
Render (GPU skinning)	3430 ms (15.4 ms/frame)
E2E	5580 ms · RTF 0.77

Reproduce: cd example && python bench_vrm.py --backend gles --model-dir .. (lipsync only) — add --body <clip.npz> for lipsync + body (see Body animation above for making a clip).

lipsync only sits at RTF ≈ 0.6, lipsync + body at RTF ≈ 0.77 — both comfortably faster than realtime. Body animation adds only the CPU FK + pose stage (4.6 ms/frame): the skinning itself runs in the vertex shader (GPU), so render cost is essentially unchanged (15 ms/frame). Render still dominates (~60–70 % of E2E) and fits the 33 ms budget of 30 fps, so a streaming setup (TTS chunk ∥ render pipeline) stays realtime. The atlas route is lighter still (RTF ~0.25): 2D sprite paste vs full 3D mesh rasterization.

Other backends (RK3576, lipsync only): numpy — pure-CPU rasterizer, ~11 s/frame (RTF ~345), a no-GPU fallback for stills only. egl (moderngl) — needs desktop GL 3.3, which ARM GPU drivers cap at 3.1 (Panfrost); x86 GPU / Mesa llvmpipe only, does not run on RK3576.

Coverage / Limitations

• Korean MFA G2P compatible — covers the phones the Ppaso G2P emits.
• ~11 unmapped phones out of 118 (loanword f/v/z, rare cʰ/dʑ/ʝ/ʁ, glide ɥ, labialized ɾʷ).
• Decomposition ratios (j-glide 30:70, w-glide 25:75) are a first pass.
• VRM route — VRM 0.x only; standard blendShapeMaster viseme presets expected. VRM has no dedicated closed-lip viseme (see README_bs2d.md).

Resources

• Lipsync module: example/ppaso_lipsync/ — visemes / extractor / keyframes / interpolate / eyes / bs2d / pipeline. API docs: example/ppaso_lipsync/README.md
• VRM render module: example/ppaso_vrm/ — loader / scene / anim / anim_body (skeleton) / raster (numpy + egl/moderngl + gles + gles_skinned).
• bs2d mapping detail: README_bs2d.md
• Examples: 07_avatar_video.py (atlas) · 08_vrm_video.py (blendshape) · 09_vrm_live.py (blendshape + body) · 10_vrm_live_streaming.py (streaming + body) · 06_streaming_with_lipsync.py · bench_lipsync.py (atlas) · bench_vrm.py (blendshape)
• Body-clip tool: example/tools/fbx_to_clip.py — FBX → .npz body clip
• Sample assets: example/avatar/ (2D) · example/avatar_vrm/ (3D VRM)

License

Apache 2.0 — same as Ppaso-TTS main.