emotion + pose → MediaPipe 478 mesh PLS

Licensing scope: See the license and provenance notice before relying on this card's license metadata for pretrained-weight redistribution or commercial use. Existing valid grants are preserved.

Current default: emotion_to_mesh_pls_v6.npz (Detectorv2 v2.8 emotion space; py-feat load_emotion_face_mesh_model() / plot_face_mesh(emotion=...)).

emotion_to_mesh_pls_v5.npz is deprecated — its column labels are permuted. Column j of v5 holds the network's j-th output (Neutral, Happy, Sad, Surprise, Fear, Disgust, Anger) but is labelled in FEAT_EMOTION_COLUMNS order (anger, disgust, fear, happiness, sadness, surprise, neutral). Indexing v5 by its labels renders the wrong expression (e.g. "anger" renders neutral). The file is retained unchanged for reproducibility only.

Each emotion one-hot vs. one-hot neutral; wireframe colored by per-vertex displacement.

Versions

file features / meshes from held-out R² CV notes
emotion_to_mesh_pls_v6.npz Detectorv2 v2.8 0.305 ± 0.006 5-fold GroupKFold by video current default, k = 27
emotion_to_mesh_pls_v5.npz Detectorv2 v2.x emotion, MediaPipe mesh 0.206 ± 0.003 3-fold GroupKFold by video deprecated (permuted labels)

Column order

feature_columns = feat.utils.FEAT_EMOTION_COLUMNS (anger, disgust, fear, happiness, sadness, surprise, neutral). Detectorv2's Fex emits Neutral, Happy, Sad, Surprise, Fear, Disgust, Anger; py-feat matches dict / Series / DataFrame inputs by name and accepts either naming. Raw arrays must be in feature_columns order.

Training data (v6)

  • 602,427 frames / 33,775 CelebV-HQ videos run through the py-feat Detectorv2 v2.8 release weights (face_multitask_v28.safetensors, md5 9e174bb3cd05ce8bed7ac64358a40332). Features and target meshes come from the same forward pass, so the map is self-consistent with what the detector emits.
  • Same frames, pose filter (|yaw| ≤ 40°, |pitch| ≤ 30°), 12-anchor GPA alignment and top-1% anchor-residual outlier drop as py-feat/au_to_mesh v6.
  • Canonicalized with an Umeyama similarity on all 478 vertices onto the au_to_mesh v4 frontal mean, the same recipe as au_to_mesh_pls_v6, so AU, emotion and blendshape renders share one frame.

Method (v6)

  • Training design [7 emotion | 3 pose | 21 pose × emotion] = 31 features; kernel PLS equivalent to PLSRegression(scale=True), k chosen by mean held-out R² (k = 27; knee at 16).
  • Deployed rows = emotion + pose (interactions vanish at pose = 0).
  • mean_neutral_mesh = prediction at one-hot neutral.
  • Per-fold R² = [0.3115, 0.3064, 0.2939, 0.3049, 0.3103].

Behaviour and limitations

Relative to one-hot neutral: happiness raises and widens the lip corners; sadness lowers the corners and raises the brows; surprise raises the brows and opens the jaw; fear widens the eyes; anger lowers the brows and narrows the inner-brow gap. All non-neutral emotions also open the mouth somewhat, reflecting the speech-heavy CelebV-HQ frames on which emotional expressions were observed. R² is modest because a 7-way softmax carries little geometric information; use this model for qualitative visualization.

Inference

from feat.plotting import plot_face_mesh, predict_face_mesh_from_features
plot_face_mesh(emotion={"Happy": 1.0})                  # Detectorv2 or py-feat names

Or with numpy only:

import numpy as np
m = np.load("emotion_to_mesh_pls_v6.npz")
cols = m["feature_columns"].tolist()
x = np.zeros(len(cols) + 3, dtype=np.float32)     # [features | Pitch, Yaw, Roll]
x[cols.index(...)] = ...
flat = x @ m["coef"] + m["intercept"]             # (1434,)
mesh = np.stack([flat[:478], flat[478:956], flat[956:]], axis=1)  # axis-major [x|y|z]

File format

NPZ: coef (k+3, 1434) f32; intercept (1434,) f32; input_columns; feature_columns (authoritative order; index by name); feature_name; pose_columns; mean_aligned_mesh (478, 3); mean_neutral_mesh (478, 3); reference_anchors (12, 3); anchor_indices (12,); n_components; model_card; training_metadata (JSON).

License and provenance

This distribution retains its existing license: mit metadata. Py-Feat's original visualization code is MIT. The historical MIT designation for this artifact does not itself settle third-party dataset or teacher rights in its fitted parameters.

The v6 artifact uses Detectorv2 v2.8 emotion and mesh outputs. The older v5 artifact uses a different detector/MediaPipe target pipeline and retains its documented label-order defect. These models use CelebV-HQ frames. The CelebV-HQ agreement restricts commercial exploitation of videos and derived data. The v2.8 detector has an ImageNet-initialized CC BY-NC 4.0 backbone checkpoint and its own dataset/teacher permission questions, summarized in the multitask notice.

Whether particular terms govern the predictions used for fitting or these PLS coefficients requires separate assessment. Review each file's training manifest and applicable agreements; do not infer either unrestricted permission or automatic transfer of every teacher restriction. Newer agreements must not be assumed to apply retroactively.

This notice clarifies scope; it does not revoke existing valid grants or create a new license for third-party material. Dataset and teacher terms do not automatically relicense every trained artifact or inference output. A software license or model-card badge alone does not establish all checkpoint redistribution or commercial-use permissions.

Downloads last month

-

Downloads are not tracked for this model. How to track
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support