Qwen2-Audio-MixAssist-LoRA

⚠ Reconstruction. The training script behind the original Qwen-Audio-Mix-Instruct release (MixAssist paper, arXiv:2507.06329) was lost. This adapter was trained with a verified reconstruction of that pipeline: everything recoverable from public artifacts is used verbatim (base model, nf4 QLoRA recipe, single epoch); the remaining hyperparameters are documented standard defaults. Behavior matches the original release in side-by-side evaluation, but weights are not bit-identical.

LoRA adapter for Qwen/Qwen2-Audio-7B-Instruct, fine-tuned on the MixAssist dataset of amateur–expert music-mixing conversations. Given a mix segment (audio) and a producer's question, it responds with concise, actionable mixing advice in the voice of the expert dialogue data.

Usage

import librosa, torch
from peft import PeftModel
from transformers import AutoProcessor, Qwen2AudioForConditionalGeneration

base = "Qwen/Qwen2-Audio-7B-Instruct"
processor = AutoProcessor.from_pretrained(base)
model = PeftModel.from_pretrained(
    Qwen2AudioForConditionalGeneration.from_pretrained(base, torch_dtype=torch.float16, device_map={"": 0}),
    "mclemcrew/Qwen2-Audio-MixAssist-LoRA",
)

audio, _ = librosa.load("my_mix_segment.wav", sr=16000)
conversation = [
    {"role": "system", "content": "You are an expert audio engineer assisting with music production and mixing. Provide clear, specific advice on audio engineering techniques, mixing adjustments, and production decisions based on the audio samples and the user's questions. Focus on practical, actionable guidance."},
    {"role": "user", "content": [
        {"type": "audio", "audio_url": "my_mix_segment.wav"},
        {"type": "text", "text": "The vocals feel buried in the chorus. What should I do?"},
    ]},
]
text = processor.apply_chat_template(conversation, tokenize=False, add_generation_prompt=True)
inputs = processor(text=text, audios=[audio], sampling_rate=16000, return_tensors="pt").to(model.device)
out = model.generate(**inputs, do_sample=True, temperature=0.7, top_k=20, top_p=0.5,
                     repetition_penalty=1.1, max_new_tokens=300)
print(processor.batch_decode(out[:, inputs["input_ids"].size(1):], skip_special_tokens=True)[0])

Decoding matters: use sampling with repetition_penalty=1.1 (the released model's own generation config). Greedy decoding without a repetition penalty produces degenerate loops on any model trained on this data.

Training

Full training code, config, and eval tooling: github.com/mclemcrew/MixAssist/tree/main/training

Setting	Value	Status
Base model	Qwen/Qwen2-Audio-7B-Instruct	ground truth
Quantization	QLoRA 4-bit nf4 + double quant, fp16 compute	ground truth
Epochs	1	ground truth (paper: 4 overfit)
Data	MixAssist, has_content==True → 241 train / 34 dev / 156 test	ground truth
LoRA	r=8, α=32, dropout 0.05, LM q/v projections only (audio tower frozen)	reconstructed default
LR / schedule	2e-4, linear, warmup 0.03	reconstructed default
Batch	1 × grad accum 8	reconstructed default

Trained in ~11 minutes on a single NVIDIA GB10; final eval loss 2.81. Audio comes from Zenodo record 15848443.

Citation

@misc{clemens2025mixassistaudiolanguagedatasetcocreative,
      title={MixAssist: An Audio-Language Dataset for Co-Creative AI Assistance in Music Mixing},
      author={Michael Clemens and Ana Marasović},
      year={2025},
      eprint={2507.06329},
      archivePrefix={arXiv},
      primaryClass={cs.SD},
      url={https://arxiv.org/abs/2507.06329},
}