Instructions to use py-feat/emotion_to_mesh with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
- Libraries
- Py-Feat
How to use py-feat/emotion_to_mesh with Py-Feat:
# No code snippets available yet for this library. # To use this model, check the repository files and the library's documentation. # Want to help? PRs adding snippets are welcome at: # https://github.com/huggingface/huggingface.js
- Notebooks
- Google Colab
- Kaggle
emotion + pose → MediaPipe 478 mesh PLS
Licensing scope: See the license and provenance notice before relying on this card's license metadata for pretrained-weight redistribution or commercial use. Existing valid grants are preserved.
Current default: emotion_to_mesh_pls_v6.npz (Detectorv2 v2.8 emotion space; py-feat
load_emotion_face_mesh_model() / plot_face_mesh(emotion=...)).
emotion_to_mesh_pls_v5.npzis deprecated — its column labels are permuted. Column j of v5 holds the network's j-th output (Neutral, Happy, Sad, Surprise, Fear, Disgust, Anger) but is labelled inFEAT_EMOTION_COLUMNSorder (anger, disgust, fear, happiness, sadness, surprise, neutral). Indexing v5 by its labels renders the wrong expression (e.g. "anger" renders neutral). The file is retained unchanged for reproducibility only.
Versions
| file | features / meshes from | held-out R² | CV | notes |
|---|---|---|---|---|
emotion_to_mesh_pls_v6.npz |
Detectorv2 v2.8 | 0.305 ± 0.006 | 5-fold GroupKFold by video | current default, k = 27 |
emotion_to_mesh_pls_v5.npz |
Detectorv2 v2.x emotion, MediaPipe mesh | 0.206 ± 0.003 | 3-fold GroupKFold by video | deprecated (permuted labels) |
Column order
feature_columns = feat.utils.FEAT_EMOTION_COLUMNS
(anger, disgust, fear, happiness, sadness, surprise, neutral). Detectorv2's Fex
emits Neutral, Happy, Sad, Surprise, Fear, Disgust, Anger; py-feat matches
dict / Series / DataFrame inputs by name and accepts either naming. Raw arrays
must be in feature_columns order.
Training data (v6)
- 602,427 frames / 33,775 CelebV-HQ videos run through the py-feat
Detectorv2 v2.8 release weights (
face_multitask_v28.safetensors, md59e174bb3cd05ce8bed7ac64358a40332). Features and target meshes come from the same forward pass, so the map is self-consistent with what the detector emits. - Same frames, pose filter (|yaw| ≤ 40°, |pitch| ≤ 30°), 12-anchor GPA alignment
and top-1% anchor-residual outlier drop as
py-feat/au_to_meshv6. - Canonicalized with an Umeyama similarity on all 478 vertices onto the
au_to_meshv4 frontal mean, the same recipe asau_to_mesh_pls_v6, so AU, emotion and blendshape renders share one frame.
Method (v6)
- Training design
[7 emotion | 3 pose | 21 pose × emotion] = 31features; kernel PLS equivalent toPLSRegression(scale=True), k chosen by mean held-out R² (k = 27; knee at 16). - Deployed rows = emotion + pose (interactions vanish at pose = 0).
mean_neutral_mesh= prediction at one-hot neutral.- Per-fold R² = [0.3115, 0.3064, 0.2939, 0.3049, 0.3103].
Behaviour and limitations
Relative to one-hot neutral: happiness raises and widens the lip corners; sadness lowers the corners and raises the brows; surprise raises the brows and opens the jaw; fear widens the eyes; anger lowers the brows and narrows the inner-brow gap. All non-neutral emotions also open the mouth somewhat, reflecting the speech-heavy CelebV-HQ frames on which emotional expressions were observed. R² is modest because a 7-way softmax carries little geometric information; use this model for qualitative visualization.
Inference
from feat.plotting import plot_face_mesh, predict_face_mesh_from_features
plot_face_mesh(emotion={"Happy": 1.0}) # Detectorv2 or py-feat names
Or with numpy only:
import numpy as np
m = np.load("emotion_to_mesh_pls_v6.npz")
cols = m["feature_columns"].tolist()
x = np.zeros(len(cols) + 3, dtype=np.float32) # [features | Pitch, Yaw, Roll]
x[cols.index(...)] = ...
flat = x @ m["coef"] + m["intercept"] # (1434,)
mesh = np.stack([flat[:478], flat[478:956], flat[956:]], axis=1) # axis-major [x|y|z]
File format
NPZ: coef (k+3, 1434) f32; intercept (1434,) f32; input_columns;
feature_columns (authoritative order; index by name); feature_name;
pose_columns; mean_aligned_mesh (478, 3); mean_neutral_mesh (478, 3);
reference_anchors (12, 3); anchor_indices (12,); n_components;
model_card; training_metadata (JSON).
License and provenance
This distribution retains its existing license: mit metadata. Py-Feat's original visualization code is MIT. The historical MIT designation for this artifact does not itself settle third-party dataset or teacher rights in its fitted parameters.
The v6 artifact uses Detectorv2 v2.8 emotion and mesh outputs. The older v5 artifact uses a different detector/MediaPipe target pipeline and retains its documented label-order defect. These models use CelebV-HQ frames. The CelebV-HQ agreement restricts commercial exploitation of videos and derived data. The v2.8 detector has an ImageNet-initialized CC BY-NC 4.0 backbone checkpoint and its own dataset/teacher permission questions, summarized in the multitask notice.
Whether particular terms govern the predictions used for fitting or these PLS coefficients requires separate assessment. Review each file's training manifest and applicable agreements; do not infer either unrestricted permission or automatic transfer of every teacher restriction. Newer agreements must not be assumed to apply retroactively.
This notice clarifies scope; it does not revoke existing valid grants or create a new license for third-party material. Dataset and teacher terms do not automatically relicense every trained artifact or inference output. A software license or model-card badge alone does not establish all checkpoint redistribution or commercial-use permissions.
