arxiv:2601.10547

HeartMuLa: A Family of Open Sourced Music Foundation Models

Published on Jan 15

· Submitted by

Dongchao Yang on Jan 16

Upvote

Authors:

Dongchao Yang ,

Yuxin Xie ,

Gongxi Zhu ,

Yuanyuan Wang ,

Haohan Guo

Abstract

A suite of open-source music foundation models is introduced, featuring components for audio-text alignment, lyric recognition, music coding, and large language model-based song generation with controllable attributes and scalable parameterization.

AI-generated summary

We present a family of open-source Music Foundation Models designed to advance large-scale music understanding and generation across diverse tasks and modalities. Our framework consists of four major components: (1) HeartCLAP, an audio-text alignment model; (2) HeartTranscriptor, a robust lyric recognition model optimized for real-world music scenarios; and (3) HeartCodec, a low-frame-rate (12.5 Hz) yet high-fidelity music codec tokenizer that captures long-range musical structure while preserving fine-grained acoustic details and enabling efficient autoregressive modeling; (4) HeartMuLa, an LLM-based song generation model capable of synthesizing high-fidelity music under rich, user-controllable conditions (e.g., textual style descriptions, lyrics, and reference audio). In addition, it provides two specialized modes: (i) fine-grained musical attribute control, which allows users to specify the style of different song sections (e.g., intro, verse, chorus) using natural language prompts; and (ii) short, engaging music generation, which is suitable as background music for short videos. Lastly, HeartMuLa improves significantly when scaled to 7B parameters. For the first time, we show that a Suno-level, commercial-grade system can be reproduced using academic-scale data and GPU resources. We expect these foundation models to serve as strong baselines for future research and to facilitate practical applications in multimodal content production.

View arXiv page View PDF Project page GitHub 151 Add to collection

Community

Dongchao

Paper author Paper submitter 1 day ago

We present a family of open-source Music Foundation Models designed to advance
large-scale music understanding and generation across diverse tasks and modalities.
Our framework consists of four major components: (1) HeartCLAP, an audiotext alignment model; (2) HeartTranscriptor, a robust lyric recognition model
optimized for real-world music scenarios; and (3) HeartCodec, a low-frame-rate
(12.5 Hz) yet high-fidelity music codec tokenizer that captures long-range musical
structure while preserving fine-grained acoustic details and enabling efficient
autoregressive modeling; (4) HeartMuLa, an LLM-based song generation model
capable of synthesizing high-fidelity music under rich, user-controllable conditions
(e.g., textual style descriptions, lyrics, and reference audio). In addition, it provides
two specialized modes: (i) fine-grained musical attribute control, which allows
users to specify the style of different song sections (e.g., intro, verse, chorus)
using natural language prompts; and (ii) short, engaging music generation, which
is suitable as background music for short videos. Lastly, HeartMuLa improves
significantly when scaled to 7B parameters. For the first time, we show that a
Suno-level, commercial-grade system can be reproduced using academic-scale
data and GPU resources. We expect these foundation models to serve as strong
baselines for future research and to facilitate practical applications in multimodal
content production.