mlx-community/Qwen2-Audio-7B-Instruct-4bit

Qwen2-Audio-7B-Instruct (4-bit MLX)

4-bit quantized version of Qwen/Qwen2-Audio-7B-Instruct for Apple Silicon via mlx-audio.

Usage

from mlx_audio.stt.utils import load_model

model = load_model("mlx-community/Qwen2-Audio-7B-Instruct-4bit")

# Transcription
result = model.generate("audio.wav", prompt="Transcribe the audio.")
print(result.text)

# Audio understanding
result = model.generate("audio.wav", prompt="What emotion is the speaker expressing?")
print(result.text)

# Translation
result = model.generate("audio.wav", prompt="Translate the speech to French.")
print(result.text)

Model Details

• Base model: Qwen/Qwen2-Audio-7B-Instruct
• Quantization: 4-bit (group_size=64), LLM only (encoder and projector kept in bf16)
• Size: ~4.2GB (vs ~15GB bf16)
• Architecture: Whisper-style encoder (32 layers) + Linear projector + Qwen2-7B LLM

Capabilities

• Speech transcription (ASR)
• Speech translation
• Audio captioning
• Emotion / sentiment detection
• Environmental sound classification
• Music understanding
• Voice chat (audio-only input)

Performance

Tested on Apple Silicon (M-series):

• ~4.7 tokens/sec generation (4-bit)
• Accurate transcription matching HuggingFace reference

Conversion

Converted using mlx-audio with:

• Audio encoder: bf16 (not quantized)
• Multi-modal projector: bf16 (not quantized)
• Language model: 4-bit quantized (group_size=64)