# Ppaso-TTS Lipsync — companion animation module

> 한국어: [README_lipsync_ko.md](./README_lipsync_ko.md) · Main: [README.md](./README.md)
>
> Blendshape mapping detail: [README_bs2d.md](./README_bs2d.md)

Pure-numpy companion module **`ppaso_lipsync`** (in `example/`) turns
Ppaso-TTS output into per-frame **2D mouth + 1D eye-blink animation curves**.
A single lipsync curve drives **two render routes**:

- **atlas** — 2D sprite swap (nearest viseme → one mouth sprite). `07_avatar_video.py`
- **blendshape** — 3D morph-target blend via `bs2d`. `08_vrm_video.py`

## Demo

**Blendshape — 3D VRM, lipsync + body idle** (`samples/08_vrm_lipsync.mp4`,
[download](https://huggingface.co/akamotaco/ppaso-tts-v1/resolve/main/samples/08_vrm_lipsync.mp4)):

<video controls src="https://huggingface.co/akamotaco/ppaso-tts-v1/resolve/main/samples/08_vrm_lipsync.mp4" width="288"></video>

**Atlas — 2D sprite** (`samples/06_lipsync_intro.mp4`,
[download](https://huggingface.co/akamotaco/ppaso-tts-v1/resolve/main/samples/06_lipsync_intro.mp4)):

<video controls src="https://huggingface.co/akamotaco/ppaso-tts-v1/resolve/main/samples/06_lipsync_intro.mp4" width="384"></video>

> 엣지디바이스용 경량화 립싱크 애니메이션 기능. 동일한 커브가 2D 스프라이트와
> 3D 블렌드쉐입 애니메이션을 모두 구동합니다.
> *(One curve drives both 2D sprite and 3D blendshape animation.)*

## How it works

```
TTS phones + durations
        │
        ▼
  viseme decompose   ← 9 anchor (A/I/U/E/O/EU/EO/M/sil) + j/w glide,
        │              palatalized / alveolar / sibilant / velar / glottal
        ▼
  keyframes → 2D xy curve     ← shared lipsync curve, x∈[-1,1] y∈[-0.1,1]
        │
        ├──────────►  atlas route      : nearest viseme → sprite swap
        │
        └──────────►  blendshape route : bs2d → barycentric → morph blend

  Eye blink (independent)  → state_timeline {0:open,1:half,2:close}
                             + curve_timeline [0,1]
```

The lipsync **curve generation** (decompose → keyframes → interpolate) is
shared. The two routes differ only in how they consume the `(x, y)` curve —
see the atlas/blendshape distinction below.

## Route A — atlas (2D sprite)

Each frame the curve point picks the **nearest viseme** and the renderer
shows that **one** mouth sprite (sprites cannot blend). Best for 2D
character art with a sprite sheet.

```bash
cd example
python 07_avatar_video.py --text "안녕하세요." --out hi.mp4
```

- Sample 2D avatar: `example/avatar/` (atlas.json + sprite sheet).
- Curve → sprite mapping: `atlas.json` `viseme_to_sprite`.

## Route B — blendshape (3D VRM)

For 3D characters with **morph-target (blendshape) visemes**. Here the curve
goes through **`bs2d`**, which finds the triangle containing the point and
returns **barycentric blend weights** — so 3 blendshapes are mixed per frame,
giving continuous mouth motion (a sprite swap cannot do this).

```
lipsync (x,y) curve → bs2d(target="vrm") → {a,i,u,e,o,neutral} weight
                    → VRM blendShapeMaster morph → mesh deform → render
```

- **vrm-spec** — bs2d maps the curve onto a standard VRM 0.x avatar's
  5 viseme + neutral expressions (Korean ㅏㅣㅜㅔㅗ → VRM a/i/u/e/o).
- Three render backends (`--backend`):
  - **gles** (default) — EGL + OpenGL ES 3.0. Real GPU acceleration on
    ARM SoCs (RK3576 Mali-G52 etc.). Needs `PyOpenGL`.
  - **egl** — moderngl + EGL desktop GL 3.3. For x86 GPUs / Mesa llvmpipe;
    ARM GPU drivers usually cap desktop GL at 3.1 so moderngl can't run
    there — use `gles` on-device. Needs `moderngl` + `glcontext`.
  - **numpy** — pure-numpy software rasterizer, zero GPU/driver dependency
    (slow — fallback only).
- Bundled sample VRM: `example/avatar_vrm/AvatarSample_F.vrm`
  (VRoid 'Vita', CC0).

```bash
cd example
python 08_vrm_video.py --text "안녕하세요." --out hi.mp4   # gles (default)
python 08_vrm_video.py --backend egl             # x86 desktop GPU
python 08_vrm_video.py --backend numpy           # no-GPU fallback
```

> **bs2d full-spec ↔ vrm-spec, the projection / barycentric stages, and how
> to map your own character's blendshapes (9 or more) — see
> [README_bs2d.md](./README_bs2d.md).**

## Body animation (optional)

`09_vrm_live.py` and `10_vrm_live_streaming.py` add a skeletal **body idle**
on top of lipsync — GPU skinning via `raster_gles_skinned`. It needs a body
**clip** (`--idle clip.npz`); **without one the body stays at rest pose**
(lipsync and video still render fine).

Make a clip from any FBX skeletal animation with `tools/fbx_to_clip.py`
(needs `bpy` — `pip install bpy`):

```bash
python tools/fbx_to_clip.py  idle.fbx  avatar_vrm/idle.npz
python 09_vrm_live.py --idle avatar_vrm/idle.npz
```

The clip is **not bundled** — animation assets such as Mixamo are
royalty-free to embed in your rendered output, but their raw motion data is
generally not redistributable, so supply your own. `BodyClipPlayer` assumes
a loop-authored clip; `fbx_to_clip.py` drops the duplicate loop endpoint.

## Quick start

```python
from ppaso_tts import PpasoTTS
from ppaso_lipsync import generate_blink_timeline

tts = PpasoTTS('./', backend='onnx')          # or 'rknn'
wav, lipsync = tts.synthesize_with_lipsync('안녕하세요.', fps=30)
# lipsync.xy_timeline   → (n_frames, 2)  — feeds atlas or blendshape route
# lipsync.keyframes     → viseme keyframes (debug)

blink = generate_blink_timeline(duration=lipsync.duration, fps=30, seed=42)
# blink.state_timeline / blink.curve_timeline
```

Streaming (chunk-to-chunk continuity is automatic):

```python
from ppaso_tts import PpasoTTS, StreamingTTS

stream = StreamingTTS(PpasoTTS('./', backend='rknn'))
for token in llm_token_stream():
    for wav, lipsync in stream.feed_with_lipsync(token):   # emits on .!?
        play_audio(wav); animate(lipsync)
for wav, lipsync in stream.flush_with_lipsync():
    ...
```

## Performance

### Atlas route — RK3576 measured

End-to-end (TTS + lipsync + eye blink + sprite composite at 768×1152, no
ffmpeg encode). RK3576, best-of-5 @ 30 fps.

| Case   | Input    | wav      | Backend        | E2E      | RTF   | Speed         |
|---     |---       |---       |---             |---       |---    |---            |
| short  | 21 chars | 3.49 s   | **RK3576 NPU** | **864 ms**  | 0.247 | **4.0× RT**   |
|        |          |          | ONNX CPU (4t)  | 897 ms   | 0.256 | 3.9× RT       |
| medium | 41 chars | 7.26 s   | **RK3576 NPU** | **1867 ms** | 0.257 | **3.9× RT**   |
| long   | 110 chars| 16.65 s  | **RK3576 NPU** | **3668 ms** | 0.220 | **4.5× RT**   |

> Reproduce: `cd example && python bench_lipsync.py --backend rknn --model-dir ..`.
> Lipsync curve generation itself adds < 1 ms on top of TTS.

### Blendshape route — RK3576 measured

End-to-end (TTS + lipsync + eye blink + morph + render at 512×768, no
ffmpeg encode — **same E2E definition as the atlas route above**).
RK3576 Mali-G52, **gles** backend, best-of-3 @ 30 fps. Two modes:
**lipsync only** (static skeleton — `08_vrm_video.py`) and **lipsync +
body** (GPU skinning + body idle animation — `09_vrm_live.py`).

**① lipsync only — E2E by text length:**

| Case   | wav      | frames | E2E        | RTF      |
|---     |---       |---     |---         |---       |
| short  | 3.51 s   | 110    | **2222 ms** | **0.63** |
| medium | 7.26 s   | 223    | **4637 ms** | **0.64** |
| long   | 16.31 s  | 497    | **9358 ms** | **0.57** |

**② lipsync + body — E2E by text length:**

| Case   | wav      | frames | E2E         | RTF      |
|---     |---       |---     |---          |---       |
| short  | 3.51 s   | 110    | **2809 ms**  | **0.80** |
| medium | 7.26 s   | 223    | **5580 ms**  | **0.77** |
| long   | 16.31 s  | 497    | **12314 ms** | **0.76** |

**③ lipsync + body — medium (7.26 s, 223 frames) stage breakdown:**

| Stage                  | medium                   |
|---                     |---                       |
| TTS + lipsync          | 1100 ms                  |
| Eye blink              | 0.5 ms                   |
| FK + pose (skeleton)   | 1031 ms (4.6 ms/frame)   |
| Morph (bs2d weight)    | 19 ms                    |
| Render (GPU skinning)  | 3430 ms (15.4 ms/frame)  |
| **E2E**                | **5580 ms · RTF 0.77**   |

> Reproduce: `cd example && python bench_vrm.py --backend gles --model-dir ..`
> (lipsync only) — add `--body <clip.npz>` for lipsync + body (see
> *Body animation* above for making a clip).
>
> **lipsync only** sits at **RTF ≈ 0.6**, **lipsync + body** at
> **RTF ≈ 0.77** — both comfortably faster than realtime. Body animation
> adds only the CPU **FK + pose** stage (~4.6 ms/frame): the skinning
> itself runs in the vertex shader (GPU), so render cost is essentially
> unchanged (~15 ms/frame). Render still dominates (~60–70 % of E2E) and
> fits the 33 ms budget of 30 fps, so a streaming setup (TTS chunk ∥
> render pipeline) stays realtime. The atlas route is lighter still
> (RTF ~0.25): 2D sprite paste vs full 3D mesh rasterization.
>
> Other backends (RK3576, lipsync only): **numpy** — pure-CPU rasterizer,
> ~11 s/frame (RTF ~345), a no-GPU fallback for stills only. **egl**
> (moderngl) — needs desktop GL 3.3, which ARM GPU drivers cap at 3.1
> (Panfrost); x86 GPU / Mesa llvmpipe only, does not run on RK3576.

## Coverage / Limitations

- **Korean MFA G2P compatible** — covers the phones the Ppaso G2P emits.
- **~11 unmapped phones out of 118** (loanword `f/v/z`, rare `cʰ/dʑ/ʝ/ʁ`,
  glide `ɥ`, labialized `ɾʷ`).
- **Decomposition ratios** (j-glide 30:70, w-glide 25:75) are a first pass.
- **VRM route** — VRM 0.x only; standard `blendShapeMaster` viseme presets
  expected. VRM has no dedicated closed-lip viseme (see README_bs2d.md).

## Resources

- **Lipsync module**: `example/ppaso_lipsync/` — visemes / extractor /
  keyframes / interpolate / eyes / **bs2d** / pipeline.
  API docs: [`example/ppaso_lipsync/README.md`](./example/ppaso_lipsync/README.md)
- **VRM render module**: `example/ppaso_vrm/` — loader / scene / anim /
  anim_body (skeleton) / raster (numpy + egl/moderngl + gles + gles_skinned).
- **bs2d mapping detail**: [README_bs2d.md](./README_bs2d.md)
- **Examples**: `07_avatar_video.py` (atlas) · `08_vrm_video.py` (blendshape)
  · `09_vrm_live.py` (blendshape + body) · `10_vrm_live_streaming.py`
  (streaming + body) · `06_streaming_with_lipsync.py`
  · `bench_lipsync.py` (atlas) · `bench_vrm.py` (blendshape)
- **Body-clip tool**: `example/tools/fbx_to_clip.py` — FBX → `.npz` body clip
- **Sample assets**: `example/avatar/` (2D) · `example/avatar_vrm/` (3D VRM)

## License

Apache 2.0 — same as Ppaso-TTS main.