ChristophSchuhmann/vocal-music-whisper

Vocal Music Whisper

Fine-tunes of laion/music-whisper specialised on vocals-only material — a cappella singing that sometimes carries a backing choir and usually does not.

The base model is a general music captioner: it describes instrumentation, tempo, genre and mood on full-band produced tracks. Confronted with dry unaccompanied voice it tends to hallucinate a backing band that is not there. These checkpoints fix that and push the description toward what actually matters in an a cappella clip: how many voices, vocal technique, register, timbre and recording character.

✅ Default checkpoint: v2/unbalanced-lr3e5 — and it is at the repo root

from transformers import WhisperForConditionalGeneration, WhisperProcessor
# no subfolder= : the repo root IS the default checkpoint
m = WhisperForConditionalGeneration.from_pretrained("ChristophSchuhmann/vocal-music-whisper")
p = WhisperProcessor.from_pretrained("ChristophSchuhmann/vocal-music-whisper")

The same weights are also kept at v2/unbalanced-lr3e5/ so the ablation stays checkable. Every other folder in this repo is an ablation or a superseded release; if you are not deliberately comparing checkpoints, use the root.

Chosen on caption quality, judged by Gemini 3.6 Flash given the audio and the caption over all 158 held-out clips, twice (full method and raw judgements): overall impression 3.47 / 5 against 3.29 for the balanced retrain, 3.23 for v1 and 2.48 for the base model.

The margin over the balanced retrain is inside the noise (+0.18, 95 % CI [-0.04, +0.39], p = 0.09) — on caption quality alone these two are a tie, and this card does not claim otherwise. The tie is broken by the axes that do separate: v2/unbalanced-lr3e5 has the better validation loss (0.9597 vs 0.9714), the better voice-gender balanced accuracy (61.6 % vs 60.6 %), and fewer factual errors flagged per caption (1.41 vs 1.71). It is behind the balanced run on nothing that was measured.

Fine-tuning did help captioning, which was not a foregone conclusion given the base model wins the gender axis outright: +0.99 overall over laion/music-whisper (CI [+0.69, +1.28], p < 1e-4). The gender regression below is real and is not cancelled by this — they are different axes and they disagree.

⚠️ Voice gender: still broken, and rebalancing the audio did not fix it

Every fine-tune here calls most female voices male. The base model does not. This was first reported on ten clips; it has now been measured on the whole 158-clip held-out split (34 female, 124 male by the Gemini audio judge), and a gender-balanced corpus was built and trained on to try to fix it. The fix did not work, and the controls say why it looked like it might.

Balanced accuracy (mean of the two recalls) is the honest summary. The split is 78 % male, so plain accuracy rewards a model for calling everything male: the previously published frozen-encoder scores 79.7 % plain accuracy while getting 31 of the 34 female clips wrong.

checkpoint	val loss	female recall	male recall	balanced acc.	abstains	mean words
base — laion/music-whisper	—	72.7 %	73.4 %	73.1 %	0.6 %	135
frozen-encoder (previous release, unbalanced)	0.9461	8.8 %	99.2 %	54.0 %	0.0 %	137
full (previous release, unbalanced)	1.1080	8.8 %	100.0 %	54.4 %	0.0 %	140
unbalanced control, lr 3e-5 ← shipped	0.9597	27.3 %	96.0 %	61.6 %	0.6 %	140
unbalanced control, lr 1e-4	0.9348	20.6 %	96.0 %	58.3 %	0.0 %	139
balanced, lr 3e-5	0.9968	11.8 %	99.2 %	55.5 %	0.0 %	141
balanced, lr 1e-4	0.9714	23.5 %	97.6 %	60.6 %	0.0 %	137

Read the controls before crediting anything. orig_* are trained on the original, unbalanced corpus on the same 10-epoch schedule as the balanced runs. The best balanced accuracy of any fine-tune is the unbalanced control orig_lr3e5 at 61.6 %, ahead of both balanced runs (60.6 % and 55.5 %). At a matched learning rate the balanced run is worse than its control (55.5 % vs 61.6 %). So the movement from the previous release's 54.0 % comes from the longer schedule with best-epoch selection, not from rebalancing the data.

Why rebalancing failed — measured, not guessed. The corpus was balanced on the audio judge's label, but the training target is the caption text, and the two are not the same thing. On the balanced set the audio is exactly 50.0 % female, yet 2,342 of its 3,189 female-voiced clips (73 %) carry a caption that calls the singer male. Balancing therefore moved the caption-text female share only from 5.6 % to 15.4 % — 2.8× — and female recall moved 8.8 % → 23.5 %, a factor of 2.7. The model tracked the label it was actually trained on, not the one the corpus was balanced on. The next attempt has to fix the captions, not the audio mix.

Until then, prefer laion/music-whisper if voice gender matters to you — it is right 73.1 % of the time on this split, better than any fine-tune here.

_{Gender is read out of the caption by a conservative token-majority rule; captions that never commit count as abstains and are excluded from the recalls, so a model cannot score well by saying less. The rule counts bass as a male token, which could favour models that talk about instruments; recomputing with bass removed leaves the ranking unchanged. The reference label is the Gemini audio judge — an instrument, not ground truth.}

v2 checkpoints (this update)

Four more frozen-encoder runs, 10 epochs each with best-epoch selection, on the unchanged 158-clip validation split. They exist to answer one question -- does balancing the corpus by voice gender fix the gender regression? -- and the answer is no. They are published because a negative result that cost 1.25 GPU-h is still a result, and because two of them are simply better than the v1 release.

folder	corpus	lr	val loss	balanced acc.	why it is here
v2/unbalanced-lr3e5 = repo root	original	3e-5	0.9597	61.6 %	the default -- best caption quality and best gender of any fine-tune
v2/unbalanced-lr1e4	original	1e-4	0.9348	58.3 %	best validation loss of any checkpoint here
v2/balanced-lr1e4	50/50 balanced	1e-4	0.9714	60.6 %	the gender-balanced retrain
v2/ablation/balanced-lr3e5	50/50 balanced	3e-5	0.9968	55.5 %	metadata only -- matched-lr control for unbalanced-lr3e5

Both v2/unbalanced-* beat the v1 frozen-encoder (val 0.9461, balanced acc. 54.0 %) on both axes, so if you are using v1 today, switch. The v1 folders stay published so the comparison remains checkable.

from transformers import WhisperForConditionalGeneration, WhisperProcessor
r = "ChristophSchuhmann/vocal-music-whisper"
m = WhisperForConditionalGeneration.from_pretrained(r)          # root = the default
p = WhisperProcessor.from_pretrained(r)
# identical weights, if you prefer to be explicit:
# WhisperForConditionalGeneration.from_pretrained(r, subfolder="v2/unbalanced-lr3e5")

Every number above is regenerated from eval/ in this repo by caption_vs_judge.py, eval_gender_val.py and make_card_v2.py; none is typed by hand.

Caption quality — how the default was chosen

The previous round of this card picked a checkpoint on voice-gender recall — one attribute. The model's job is to write a good caption, and a checkpoint can win on gender agreement while writing worse captions, so caption quality was measured separately and it is what selects the default.

Method. Every checkpoint captioned all 158 held-out clips (the unchanged validation split; greedy decoding, 440 new tokens — the most Whisper's 448-position decoder allows once its three forced start tokens are counted). Gemini 3.6 Flash was then given the audio and one caption and asked to score that caption on three axes, every level anchored in words rather than left as a bare 0–5 scale. The judge is blind: it never learns which checkpoint wrote the caption, never sees a second caption, and never sees the reference. It must write a one-line "what I hear" and list the specific statements it believes are wrong before scoring, so a surprising number can be traced to the perception behind it.

axis	question
detail 0–5	how specific and informative, versus vague boilerplate. Wrong facts are not penalised here, and length alone earns nothing.
factual correctness 0–5	does it match what is actually audible — voice type, solo vs layered, instruments present or absent, genre, mood. This is why the audio is supplied.
overall impression 0–5	the caption as a whole. Not an average; usefulness weighed against trustworthiness.

Every clip was judged twice independently (1,264 judgements in total, 0 failures, every response asserted to carry non-zero AUDIO prompt tokens) and the two passes are averaged, which halves the judge's own noise. Raw judgements: eval/capjudge.jsonl; summary: eval/capjudge_summary.json.

checkpoint	detail	factual	overall	s.e.m.	errors flagged / caption	captions with none
base — laion/music-whisper	3.62 ± 0.50	2.57 ± 1.48	2.48 ± 1.38	0.11	2.42	15 %
v1 frozen-encoder	4.87 ± 0.29	3.21 ± 1.35	3.23 ± 1.35	0.11	1.65	23 %
v2 balanced lr 1e-4	4.86 ± 0.28	3.27 ± 1.45	3.29 ± 1.44	0.11	1.71	29 %
v2 unbalanced lr 3e-5 ← default	4.91 ± 0.25	3.45 ± 1.43	3.47 ± 1.41	0.11	1.41	34 %

n = 158 clips per checkpoint, each judged twice. Spread is the standard deviation across clips — it is large because clips differ enormously in how hard they are to caption, which is exactly why the comparisons below are paired.

The two v2 contenders are within noise of each other

comparison	overall Δ	95 % CI (bootstrap, paired)	p	clips won / tied / lost	verdict
v2/unbalanced-lr3e5 − v2/balanced-lr1e4	+0.18	[-0.04, +0.39]	0.09	64 / 43 / 51	tie — do not read a winner into this
v2/unbalanced-lr3e5 − v1 frozen-encoder	+0.24	[+0.02, +0.47]	0.04	63 / 39 / 56	separated
v2/unbalanced-lr3e5 − base	+0.99	[+0.69, +1.28]	<1e-4	96 / 18 / 44	separated

Three things this evaluation says that the gender evaluation could not:

1. Fine-tuning helps captioning, clearly. Every fine-tune beats the base model by roughly three-quarters of a point of overall impression. That is worth stating plainly because the base model beats every fine-tune on voice gender — the two axes genuinely disagree, and only measuring both makes that visible.
2. The two v2 contenders cannot be separated on caption quality. The gap between them (+0.18) is smaller than the judge's own test–retest disagreement (mean absolute difference 0.56 points on overall, 63 % exact agreement between the two passes). The default goes to v2/unbalanced-lr3e5 on the tie-breakers listed at the top of this card, not on this number.
3. Detail is at ceiling and factual correctness is the whole story. All three fine-tunes score 4.86–4.91 on detail with 135 of 158 clips at the maximum: they have all learned to write richly specific prose, and detail no longer discriminates between them. What separates them from the base model, and what would separate a future checkpoint from these, is whether the specifics are true.

Read this as a relative measurement, not an absolute grade. The judge is one model listening to synthetic singing; on some clips it hears an instrument the reference caption says is not there. That bias applies identically to all four checkpoints on identical audio, which is what makes the paired differences meaningful and the absolute values only indicative.

Voice gender is unchanged by the longer generation. Regenerating the captions at 440 tokens instead of 200 moves the balanced accuracy of all four checkpoints by 0.0 percentage points, so the gender table below stands as published.

🎧 Five held-out clips, captioned by the default checkpoint

Each clip was held out of training. Press play, then read what v2/unbalanced-lr3e5 says about it and what the judge scored that exact caption. The last one is a failure, included because five flattering clips would make a prettier card and a useless one.

g60_bedroom_pop_030 — bedroom pop

▶ play / download the clip

v2/unbalanced-lr3e5 says: This recording features a solo female voice performing an intimate indie-pop melody without any instrumental accompaniment. The singer employs a breathy, delicate delivery with a light mixed voice that occasionally breaks into a delicate falsetto for higher melodic peaks. Her timbre is warm and resonant yet possesses a fragile, airy quality that enhances the song's melancholic and nostalgic mood. The vocal range is primarily in the mezzo-soprano register, moving fluidly between a grounded chest voice and a light, airy head voice. The phrasing is predominantly legato, with subtle glottal onsets and a narrow, controlled vibrato that adds a sense of vulnerability to the performance. The recording is dry and close-miked, capturing the fine details of the vocal texture, with a light touch of reverb that provides a sense of space without masking the detail. The overall expressive arc is one of quiet reflection and longing, maintaining a consistent, intimate character throughout the clip.

Judge: detail 5.0, factual 5.0, overall 5.0  ·  what it heard: A solo female voice sings a gentle indie-pop song softly without any instrumental accompaniment.  ·  An excellent, highly descriptive, and accurate caption that perfectly fits the audio clip.

choir_1aad9c13bb_s5054 — choir

▶ play / download the clip

v2/unbalanced-lr3e5 says: A solo male baritone performs an intimate, a cappella folk melody with a warm and resonant timbre. The vocal delivery is characterized by a breathy, mixed-voice technique and a gentle, controlled vibrato that adds a sense of vulnerability. The phrasing is predominantly legato, with subtle glottal attacks and a slight breathiness that enhances the emotional intimacy of the performance. The recording is dry and close-miked, capturing the fine details of the singer's breath and articulation without any instrumental accompaniment or digital effects. The overall mood is one of quiet reflection and longing, maintained through a consistent expressive arc.

Judge: detail 5.0, factual 5.0, overall 5.0  ·  what it heard: A solo male baritone sings an a cappella folk-style song in a warm, intimate tone with no instrumental accompaniment.  ·  The caption is highly detailed, entirely accurate, and paints a precise picture of the audio recording.

g60_americana_singer_songwriter_011 — americana singer songwriter

▶ play / download the clip

v2/unbalanced-lr3e5 says: This recording features a solo male baritone performing a dramatic musical theater piece with a powerful, resonant delivery. The singer employs a strong mixed voice and belting technique, particularly in the higher register where the tone becomes more intense and slightly gritty. His phrasing is primarily legato, punctuated by clear glottal attacks and a controlled, medium-width vibrato that adds emotional weight to the sustained notes. The timbre is warm and full-bodied, conveying a sense of yearning and determination that builds throughout the clip. The recording is entirely unaccompanied, with no instrumental backing or background noise, and the sound is dry and close-miked with a subtle touch of reverb that enhances the natural resonance of the voice.

Judge: detail 5.0, factual 4.0, overall 4.0  ·  what it heard: A male baritone sings a theatrical piece solo and unaccompanied.  ·  The caption is exceptionally detailed and completely accurate, offering a precise description of the audio clip.

g60_metal_symphonic_soprano_019 — metal symphonic soprano

▶ play / download the clip

v2/unbalanced-lr3e5 says: This recording features a solo male voice that begins with a gentle, narrative quality, eventually joined by layered backing vocals and lush harmonies that create a rich, choral texture. The performance sits firmly within the contemporary folk or musical theater genre, characterized by its storytelling approach and clear, articulate diction. The singer employs a smooth legato phrasing with a controlled, medium-width vibrato that adds warmth to the sustained notes. The vocal timbre is resonant and warm, possessing a pure, pure quality that remains consistent across a baritone-tenor range. The expressive arc moves from a soothing, lullaby-like opening to a more rhythmic and harmonically dense conclusion. This is an entirely unaccompanied vocal performance with no audible instrumentation or background noise. The recording is polished and dry, with a subtle room reverb that provides a sense of space while maintaining a close, intimate feel.

Judge: detail 5.0, factual 3.0, overall 3.0  ·  what it heard: I hear an unaccompanied solo female voice singing a lullaby-like folk song.  ·  flagged: solo male voice; baritone-tenor range  ·  While highly detailed and descriptive of the musical texture, the incorrect gender identification misleads the listener on a core element.

g60_melodie_francaise_044 — melodie francaise

▶ play / download the clip

v2/unbalanced-lr3e5 says: This recording features a solo male voice that begins with a conversational, intimate delivery before being joined by layered backing vocals and lush harmonies that create a powerful, anthemic sound. The genre is firmly rooted in Indie Folk, characterized by a warm, resonant timbre and a slight grit that adds emotional weight to the performance. The singer demonstrates a versatile range, moving from a breathy lower register to a soaring, belted mixed voice in the higher passages. The vocal technique includes subtle glottal attacks and a controlled vibrato that widens during sustained notes. The expressive arc is one of building intensity, starting with a quiet, nostalgic reflection and culminating in a passionate, yearning climax. The recording is entirely unaccompanied, featuring only the human voice with no instrumental backing or background noise. The production is polished and dry, with a close-miked quality that is enhanced by a touch of studio reverb, providing a sense of space without masking the detail of the vocal performance.

Judge: detail 4.0, factual 1.0, overall 1.0  ·  what it heard: I hear a solo female voice singing an pop/indie-country style phrase without any instrumental accompaniment or backing vocals.  ·  flagged: solo male voice; joined by layered backing vocals and lush harmonies; soaring, belted mixed voice in the higher passages; slight grit  ·  Despite being richly descriptive, the caption completely misidentifies the primary voice gender and invents non-existent layered backing vocals.

🎧 Listen — interactive demo

Ten held-out clips, three captions each →

Play each clip, then read what the base model, the full fine-tune and the frozen-encoder checkpoint each say about it, with the Gemini reference caption underneath. The ten span genre families and the solo / backing / instrument flags including cases the models get wrong — ten flattering clips would make a prettier page and a useless one.

Measured on those ten:

• The base model describes, but it describes the wrong thing. The claim that once stood here — nine of ten base outputs are a lyric transcript — was measured against openai/whisper-small, which is not the base of anything in this repo, and is withdrawn. Measured on all 158 held-out clips, 158 of 158 laion/music-whisper outputs (100 %) are music description; 0 are lyric transcripts and 0 are degenerate. The problem is that it describes an imagined band — tempo, drums, production — over clips that are one unaccompanied voice.
• Voice gender is right in 9 of 10. The miss is opera_soprano_coloratura.
• One clip exposes a generation failure rather than a captioning one: on metal_symphonic_soprano the prompt asked for a female voice, and the reference caption and both fine-tunes independently hear a male tenor. LeVo 2 ignored the gender token; the captioners are right.

The two checkpoints

Both live in this repo as subfolders. Load with subfolder=:

from transformers import WhisperProcessor, WhisperForConditionalGeneration

sub = "full"          # or "frozen-encoder"
proc = WhisperProcessor.from_pretrained("ChristophSchuhmann/vocal-music-whisper", subfolder=sub)
model = WhisperForConditionalGeneration.from_pretrained("ChristophSchuhmann/vocal-music-whisper", subfolder=sub)

	full	frozen-encoder
What moves	encoder and decoder	decoder only, encoder frozen
Trainable params	241M / 242M	154M / 242M
Learning rate	1e-05	5e-05
Result	val loss 3.2411 → 1.1080 (65.8% lower), 0.13 GPU-h	val loss 3.2411 → 0.9461 (70.8% lower), 0.14 GPU-h

Superseded — these are the v1 checkpoints. The default is now the repo root (v2/unbalanced-lr3e5), which beats both of these on caption quality, validation loss and voice gender. The comparison below is kept because it is why every later run freezes the encoder.

Of the two v1 checkpoints, frozen-encoder is the better one. full adapts the acoustic front end as well as the caption language model, so it fits dry unaccompanied voice best — but it is also the one that can drift away from the base model's general music knowledge. frozen-encoder cannot forget anything the encoder knows, by construction: the audio representation is bit-identical to the base model's and only the decoder's captioning behaviour changes. If you intend to caption a mix of a cappella and full-band music with a single model, prefer frozen-encoder. If you only ever see solo voice, full is the better fit.

Why these learning rates

Whisper-small supervised fine-tuning conventionally sits at 1e-5 (HF's Whisper fine-tuning recipes use 1e-5 down to 6.25e-6; Whisper's own paper pretrains small at ~5e-4, an order of magnitude above any fine-tune). On a corpus of this size, 5e-5+ applied to the whole model is enough to overwrite the general music-captioning behaviour we explicitly wanted to keep — so full uses 1e-5 with 10% warmup and cosine decay, which is the standard "adapt without erasing" setting.

frozen-encoder uses 5e-5 precisely because far fewer parameters are in play: with the encoder held fixed, the same nominal LR moves a much smaller, better-conditioned subspace, and the risk it carries is decoder overfitting rather than representational collapse. Same-LR comparisons between a full and a frozen fine-tune are not apples-to-apples; matching each to its own parameter count is.

Training data

Two datasets, combined and then downsampled to balance:

	clips	female	male	female share
TTS-AGI/levo2-vocals-captioned	8,582	1,483	7,090	17.3 %
TTS-AGI/levo2-vocals-gender-rebalance	4,800	1,740	3,057	36.3 %
balanced training set	6,378	3,189	3,189	50.0 %

All clips are LeVo 2 gen_type: vocal generations captioned by Gemini 3 Flash from the audio, with every call asserted to carry a non-zero AUDIO prompt-token entry so no fabricated caption could enter the set.

The fix that motivated the retrain

The male skew was not the captioner. A controlled probe (36 clips per arm, same genres and seeds, only the position of the gender word changed) found it is prompt word order in the generator: with the gender word first, 11.1 % of clips come out female; with the same word last, 47.2 %. Confirmed acoustically at median F0 324 Hz against 276 Hz.

4,800 new clips were generated with the winning gender_last variant. The honest yield was 36.3 %, not the 47.2 % the small probe suggested — a second, larger probe of the same variant measured 36.1 %, which is what the full run reproduced. The prompt fix roughly doubles female yield and does not solve the problem outright.

The balanced set was then drawn from the combined pool with no oversampling (every clip appears at most once), by stratified downsampling of the male side genre by genre with a floor of 12 per genre, so all 85 genres survive. Male clips were selected by deterministic id hash, so the choice is reproducible and uncorrelated with caption length or quality. Clips the judge could not gender were dropped rather than assigned a side.

Two instruments, and they disagree. The audio judge and a caption-derived gender label agree per clip 84.8 % of the time yet differ ~3× on the corpus-wide female base rate. The balance above is by the audio judge. Treat both as instruments, not ground truth.

The validation split

158 clips, held out from every genre bucket with a fixed seed before any training, and unchanged across both rounds — so every val loss on this page is comparable to the 3.2411 base figure and to the previous release. No newly generated clip entered it: the val split predates the rebalancing run and is 22 % female, which is why balanced accuracy rather than raw accuracy is the headline.

Hyperparameters

	full	frozen-encoder
Base	laion/music-whisper (whisper-small topology, 12+12 layers, d=768)	same
Epochs	3	3
Batch (per GPU × GPUs)	8 × 1 = 8	8 × 1 = 8
Optimizer	AdamW β=(0.9, 0.98), eps 1e-6, wd 0.01	same
Schedule	cosine, 10% linear warmup	same
Grad clip	1.0	same
Precision	bf16 autocast	same
Max target tokens	448	same
GPU-hours	0.131	0.136
Hardware	1x GH200 (aarch64) per run; 4 runs concurrently on one node	same

Loss curves

full — train: 25:3.255 · 300:1.686 · 600:1.320 · 875:1.242 · 1150:1.142 · 1450:1.126 · 1725:1.071 · 2025:1.097 · 2300:1.050 · 2575:1.024 · 2875:1.019 · 3150:1.034

full — val: 0:3.241 · 1053:1.217 · 2106:1.121 · 3159:1.108

frozen-encoder — train: 25:3.173 · 300:1.316 · 600:1.096 · 875:1.055 · 1150:0.930 · 1450:0.924 · 1725:0.867 · 2025:0.888 · 2300:0.792 · 2575:0.775 · 2875:0.765 · 3150:0.772

frozen-encoder — val: 0:3.241 · 1053:1.033 · 2106:0.950 · 3159:0.946

(step : loss)

Data-mix ablation

Gemini flagged a substantial minority of clips as carrying audible instruments despite every prompt asking for none, so "what is actually in the training mix" is a real axis rather than a cosmetic one. The same two variants were therefore also trained on the subset with instruments_audible == false. The validation set is identical and unfiltered in every cell, so all four val losses are directly comparable.

Run	Variant	Mix	Train clips	Val before	Val (best, published)	Best epoch	GPU-h
full_all	full	all	8,424	3.2411	1.1080	3	0.131
frozen_all	frozen	all	8,424	3.2411	0.9461	3	0.136
full_vocalsonly	full	vocalsonly	5,431	3.2411	1.2605	3	0.092
frozen_vocalsonly	frozen	vocalsonly	5,431	3.2411	1.0992	2	0.074

Two results fall out of this, and the second was not the expected one:

1. Freezing the encoder wins. frozen-encoder reaches a lower val loss than the full fine-tune on both data mixes, while being structurally incapable of forgetting what the encoder knows about general music.
2. Filtering the training data to "vocals only" makes it worse, not better. Dropping the 2,993 clips Gemini flagged as carrying audible instruments costs roughly 0.15 val loss in both variants. The val set is unfiltered and reflects the real distribution, so a model that never saw a backing instrument is simply worse at describing the clips that have one. The intuitive clean-up hurt.

Before / after on held-out val clips

tr_bulgarian_diaphonic_evo_s2 — genre bulgarian_diaphonic

Gemini reference: This recording features a solo male tenor voice performing a pop-rock arrangement, accompanied by layered vocal harmonies during the chorus sections. The singer begins the verse in a warm, resonant mid-register with a legato phrasing and a relatively straight tone, punctuated by subtle glottal attacks for emphasis. As the track progresses into the chorus, the vocal style shifts towards a powerful belt, utilizing a well-support…

Base laion/music-whisper: The listener hears a recording dominated by a male vocalist. The vocal performance is characterized by a clear and slightly nasal timbre, conveying a sense of sincerity. The vocal delivery is the primary focus, with no other non-lyrical vocal sounds present. The song's instrumentation is minimal, consisting solely of a simple acoustic guitar accompaniment. The tempo is slow, contributing to the overall mood. The genre is class…

full: This recording features a solo male tenor voice that begins with an intimate, breathy delivery before transitioning into a powerful, multi-layered vocal arrangement. The genre is firmly rooted in Indie Pop, characterized by its soulful and uplifting emotional arc. The singer employs a warm, resonant timbre with a slight grit that adds a sense of vulnerability to the performance. As the track progresses, the vocalist transition…

frozen-encoder: This recording features a solo male tenor voice that begins with an intimate, breathy delivery before expanding into a rich, multi-tracked arrangement with layered harmonies and doubled vocal lines. The genre is contemporary pop-rock, characterized by a soulful and emotive vocal style. The singer employs a mix of chest and head voice, transitioning into a powerful belt during the more intense passages. His technique includes s…

levo2__metal_s0 — genre metal

Gemini reference: This recording features a powerful male tenor performing an anthemic arena rock passage, beginning with a gritty solo vocal that eventually expands into a layered, multi-part harmony. The singer employs a robust chest-dominant mix and high-energy belting, characterized by a warm yet textured timbre with a natural rasp that adds emotional weight. His delivery is marked by sustained legato lines and deliberate glottal attacks, n…

Base laion/music-whisper: The listener hears a track that immediately establishes a strong rhythmic foundation with a driving drum beat and a prominent bassline. The tempo is approximately 120 beats per minute, contributing to the song's energetic feel. The instrumentation is primarily electronic, featuring a distorted synth bass providing a heavy low-end presence, alongside a driving drum beat that propels the music forward. A male vocalist takes the …

full: This recording features a powerful male tenor performing in a dramatic musical theatre style, beginning with a solo lead that is later joined by lush, multi-tracked backing harmonies. The singer employs a robust mixed-voice technique, transitioning into a resonant belt with a controlled, medium-width vibrato and occasional glottal attacks for emotional emphasis. His timbre is warm and rich, characterized by a slight grit and a…

frozen-encoder: This recording features a powerful male tenor performing in a contemporary alternative rock style, beginning with a solo vocal that soon expands into a rich, multi-layered arrangement with doubled voices and soaring harmony parts. The singer employs a robust mixed-voice technique, transitioning into intense belting with a controlled, medium-width vibrato and occasional glottal attacks that add emotional weight. His timbre is w…

g60_metal_black_018 — genre metal_black

Gemini reference: This high-energy alternative rock performance features a powerful solo male lead complemented by rich, multi-tracked backing harmonies that provide a dense, anthemic texture. The vocalist utilizes a robust belting technique with a gritty, distorted edge, characteristic of post-grunge aesthetics. His delivery is marked by a fast, wide vibrato and aggressive glottal attacks, transitioning seamlessly between a resonant chest-domi…

Base laion/music-whisper: The listener hears a track that immediately establishes a strong rhythmic foundation with a driving drum beat and a prominent bassline. The tempo is approximately 140 beats per minute, contributing to the song's energetic feel. The instrumentation is characterized by a prominent drum kit, featuring a punchy kick drum, a snare drum, and a hi-hat, providing a solid rhythmic backbone. A male vocalist takes the lead, delivering th…

full: This recording features a powerful male solo voice that transitions into a rich, multi-layered arrangement with layered backing harmonies and unison doubling. The genre is a dramatic musical theater-rock, characterized by a defiant and intense delivery. The singer employs a robust mixed voice with a resonant, warm timbre, utilizing a strong chest-dominant mix and occasional glottal attacks for emotional emphasis. The vocal ran…

frozen-encoder: This recording features a powerful male solo voice that transitions into a layered, anthemic arrangement with doubled vocals and harmonies during the climactic sections. The performance sits firmly within the alternative rock genre, characterized by its defiant and anthemic energy. The singer employs a robust belting technique, utilizing a strong mixed voice to reach the higher register with a slight, intentional grit. His tim…

g60_opera_tenor_dramatic_056 — genre opera_tenor_dramatic

Gemini reference: This recording features a solo male lead, likely a high baritone or tenor, who begins with a soulful, contemplative delivery before being joined by a powerful, multi-layered gospel-style choir. The genre leans heavily into contemporary gospel with folk-rock sensibilities, characterized by a transition from intimate storytelling to a triumphant choral anthem. The lead vocalist employs a resonant chest-mix with a moderate vibrat…

Base laion/music-whisper: The listener hears a piece of music primarily characterized by a female vocalist. The vocal performance is delivered by an adult female, exhibiting a clear and slightly operatic timbre. The vocal quality is described as clear and resonant. The instrumentation is composed of a piano, strings, and a choir. The tempo of the music is slow, approximately 70 beats per minute. The genre of the music is classified as either Classical …

full: This recording features a powerful male tenor performing in a dramatic musical theater style, beginning with a solo voice that is later joined by lush, multi-tracked backing harmonies. The singer employs a resonant mixed voice with a warm, slightly gritty timbre, transitioning into a powerful belt as the melody ascends. His technique is characterized by a wide, controlled vibrato and precise legato phrasing, punctuated by occa…

frozen-encoder: This recording features a powerful male tenor performing in a contemporary musical theater style, beginning with a solo vocal that soon expands into a rich, multi-layered arrangement with lush harmonies and doubled parts. The singer employs a resonant mixed voice with a bright, forward placement, utilizing a consistent vibrato and smooth legato phrasing that transitions into a powerful belt in the upper register. The timbre is…

What improved

• No more phantom instruments. The base model reliably invents a band — "gentle acoustic guitar", "soft piano accompaniment", "steady drum groove" — under clips that are literally one unaccompanied voice. Both fine-tunes stop doing that on this domain.
• Voice-count awareness. The captions now distinguish a genuinely solo take from one with doubled/harmonised/choir backing, which is the single most useful label in a cappella data and the axis the dataset was built around.
• Vocal vocabulary. Register, tessitura, vibrato width, belt vs mixed vs head voice, glottal onsets, melisma — the base model rarely reaches for these; the fine-tunes do.
• Format consistency. Output is flowing prose of predictable length rather than the base model's mix of prose and "well-suited for…" ad copy.

What regressed

Reported honestly, because both of these are real:

• Instrumentation description is degraded outside this domain. The training targets almost never contain instruments, so full in particular has learned that instruments are usually absent. On full-band music it under-describes or omits the backing. frozen-encoder is markedly more robust here — the encoder is untouched — which is why it is the safer general pick.
• Genre calls inherit LeVo 2's failure modes. The training audio is synthetic, and for nine genres LeVo 2 provably cannot produce the requested style (see the dataset card). Captions for those clips describe what was generated, not what was asked for. The models will reproduce that mapping.
• Synthetic-audio bias. Every training clip is a neural generation. Expect some transfer gap to real human a cappella recordings; the recording-character sentences in particular describe LeVo 2's rendering characteristics.
• Tempo/BPM and key statements are less reliable. They were rarely present in the targets, so they got no reinforcement.

Intended use

Captioning and tagging of solo-voice / a cappella audio: dataset curation, retrieval, filtering for singing-voice corpora. Use the repo root unless you are deliberately comparing checkpoints. Not a general music captioner — use the base laion/music-whisper for that, and use it too if voice gender is what you need.

Reproducing

Everything below is in code/ in this repo and runs as written.

Caption your own audio — files, globs or a directory; the default checkpoint is the repo root, so no --subfolder is needed:

pip install "transformers>=4.40" torch soundfile scipy
python code/caption.py song.wav
python code/caption.py clips/ --batch 8 --jsonl captions.jsonl
python code/caption.py song.wav --model ChristophSchuhmann/vocal-music-whisper --subfolder v2/balanced-lr1e4   # an ablation

or in four lines of Python:

import soundfile as sf
from transformers import WhisperForConditionalGeneration, WhisperProcessor

proc = WhisperProcessor.from_pretrained("ChristophSchuhmann/vocal-music-whisper")
model = WhisperForConditionalGeneration.from_pretrained("ChristophSchuhmann/vocal-music-whisper").eval()
model.config.forced_decoder_ids = None            # or it transcribes lyrics instead
model.generation_config.forced_decoder_ids = None

x, sr = sf.read("song.wav")                        # must be mono 16 kHz
feats = proc(x, sampling_rate=16000, return_tensors="pt").input_features
print(proc.batch_decode(model.generate(feats, max_new_tokens=440, num_beams=1),
                        skip_special_tokens=True)[0])

max_new_tokens=440 matters: these captions run 130–190 words, and the 200-token cap used in an earlier revision cut a third of them off mid-sentence. 440 is the ceiling — Whisper's max_target_positions is 448 and the three forced start tokens count against it.

Train it. One GPU per run; the four v2 runs fit on one 4-GPU node (code/retrain.sbatch is the exact Slurm script, including the 150-second stagger between launches that stops concurrent import torch off a shared filesystem from wedging every worker):

python code/finetune.py --variant frozen --mix all --lr 3e-5 \
    --data train.jsonl --val val.jsonl --epochs 10 --bs 8 \
    --tag unbalanced-lr3e5 --out ckpt/unbalanced-lr3e5      # <- the default checkpoint

Regenerate the held-out captions (code/capgen.sbatch runs all four checkpoints across four GPUs):

python code/eval_gender_val.py --name orig_lr3e5 --path ckpt/unbalanced-lr3e5 \
    --max-new-tokens 440 --out capgen_orig_lr3e5.json

Re-judge caption quality (needs a Gemini key in HYPRLAB_API_KEY; ~$5 and ~10 minutes for all 1,264 judgements, and it resumes from the JSONL if interrupted):

python code/judge_caption.py \
    --preds base=capgen_base.json frozen_all_3ep=capgen_frozen_all_3ep.json \
            orig_lr3e5=capgen_orig_lr3e5.json bal_lr1e4=capgen_bal_lr1e4.json \
    --out capjudge.jsonl --workers 16 --rep 0
python code/judge_caption.py ... --rep 1            # second independent pass
python code/capjudge_summary.py --jsonl capjudge.jsonl --out capjudge_summary.json

Re-measure voice gender on the same split:

python code/eval_gender_val.py --name orig_lr3e5 --path ckpt/unbalanced-lr3e5
python code/evalg_summary.py

Related

The generator. The audio is LeVo 2 / SongGeneration v2. laion/moss-tts-local-transformer-4.55b-voice-acting-v2 is not the base model for anything here — that is a speech/voice-acting model and is unrelated to this music corpus.

LeVo 2 weights used (gen_type: vocal)	https://huggingface.co/lglg666/SongGeneration-v2-large
LeVo model collection	https://huggingface.co/collections/lglg666/levo
LeVo technical report (arXiv:2506.07520)	https://arxiv.org/abs/2506.07520
LeVo 2 audio samples	https://levo-demo.github.io/levo_v2_demo/

This project.

Dataset	https://huggingface.co/datasets/TTS-AGI/levo2-vocals-captioned
Fine-tuned captioner	https://huggingface.co/ChristophSchuhmann/vocal-music-whisper
Listening pages (per-genre players + scores)	https://tts-agi-moss-voice-profiles.static.hf.space/vocals/genres.html
Genre space	https://tts-agi-moss-voice-profiles.static.hf.space/genre_space.html
MOSS voice-acting manual	https://laion-ai.github.io/moss-voiceacting-manual/site/index.html

Scoring and captioning models.

laion/music-whisper (captioner base, score encoder)	https://huggingface.co/laion/music-whisper
laion/music-aesthetics	https://huggingface.co/laion/music-aesthetics
laion/music-popularity-full-ft	https://huggingface.co/laion/music-popularity-full-ft
Captioner	Gemini 3 Flash, native generateContent endpoint

Citation

@misc{vocalmusicwhisper,
  title  = {Vocal Music Whisper: a cappella-specialised music captioning},
  author = {LAION},
  year   = {2026},
  url    = {https://huggingface.co/ChristophSchuhmann/vocal-music-whisper}
}

Base model laion/music-whisper (CC-BY-4.0). Training data TTS-AGI/levo2-vocals-captioned. Released under CC-BY-4.0.