API Try Docs Blog

LFM3-Omni-2B-ONNX

ONNX version of LFM3-Omni-2B for cross-platform inference with ONNX Runtime.

Find architecture and usage details in the original model card.

Files

---

Each precision pairs a decoder with a vision encoder, an audio encoder and a codec decoder. Every precision also runs the float32 depth_decoder, which the sizes include.

Precision	Decoder	Vision encoder	Audio encoder	Codec decoder	Size	Device
q4	decoder_q4	vision_encoder_q8	audio_encoder	codec_decoder	2.91 GB	CPU
q8	decoder_q8	vision_encoder_fp16	audio_encoder	codec_decoder	3.73 GB	CPU
q4f16	decoder_q4f16	vision_encoder_fp16	audio_encoder_fp16	codec_decoder_fp16	2.47 GB	GPU
q8f16	decoder_q8f16	vision_encoder_fp16	audio_encoder_fp16	codec_decoder_fp16	3.12 GB	GPU
fp16	decoder_fp16	vision_encoder_fp16	audio_encoder_fp16	codec_decoder_fp16	4.68 GB	GPU
fp32	decoder	vision_encoder	audio_encoder	codec_decoder	9.03 GB	Reference

The fp16, q4f16 and q8f16 decoders run with fp16 activations, the others with fp32. q4 and q4f16 keep the embedding, the LM head and 27 of the 153 layer projections in int8 and the rest in int4. Every decoder keeps the speech-frame codebook table in float.

Use with ONNX Runtime

---

The vision encoder maps each image, or each temporal patch of a video, to projected features, and the audio encoder maps the log-mel features of 16 kHz audio to audio features. The decoder takes them at the <|image|>, <|video|> and <|audio|> placeholders of input_ids (image_features, video_features, audio_features), and the codes of generated speech frames at the <|audio_out|> placeholders (audio_out_codes). Each step also takes attention_mask, position_ids, logits_to_keep and the convolution and attention caches (past_conv.N, past_key_values.N.key and .value). It returns logits, the final-norm hidden_states and the updated caches (present_conv.N, present.N.key and .value). The fp16, q4f16 and q8f16 decoders keep their caches in fp16.

For speech, the depth decoder predicts the 12 codes of each speech frame from the decoder's hidden state in 3 MaskGIT passes, and the codec decoder maps the codes to STFT features that an inverse STFT turns into 24 kHz audio. For preprocessing and the inverse STFT, use the values in processor_config.json and audio_tokenizer/config.json.

On GPU, use fp16, q4f16 or q8f16. The CUDA execution provider has no fp32 GroupQueryAttention kernel, so the other graphs run attention on the CPU.

In the browser, use import * as ort from 'onnxruntime-web/webgpu': the default onnxruntime-web import cannot run these graphs. On WebGPU, pair decoder_q4f16 with the float32 side graphs of the q4 bundle (vision_encoder_q8, audio_encoder, codec_decoder), and prefill in chunks of at most 1,024 tokens.

Recommended sampling: greedy decoding for text, and audio_temperature=0.4, audio_repetition_penalty=1.2, audio_repetition_window=64 for speech or interleaved output. More options in the ONNX guide.