Struggling to balance Melody Accuracy vs. Natural Vocals on Low-Pitched Vocal in high gain instrument Songs
Has anyone else noticed that the model struggles when the reference vocal is male or female has a deep, soft, thin voice tone? When generating a cover, the melody often feels "loose" or inaccurate. To compensate, I have to significantly increase the cover or cover noise parameters. However, pushing these settings too high causes the result to sound like a rigid copy of the original reference track. This excessive mimicry makes the output sound like lip-syncing, which completely ruins the ability to change lyrics to a different language or add new lyrics, as the model is locked into the original performance rather than generating a flexible vocal.
It feels like there is no "sweet spot" for these types of songs. Is this a known bottleneck for the current model? Does anyone have a workflow that allows for accurate melody alignment without forcing the output to sound like a perfect, lifeless copy of the source that prevents lyric adaptation?
Normal female high vocal (9/10)
Struggling case (5/10 - so random that this is the best value I could set):
The model cannot do everything, it's a matter of chance in the distribution. Could you share the source audio and the full generation JSON? I'll check with XL SFT / XL merges and XL Turbo
Struggling case KimiNiTodoke - TomofumiTanizawa
""""
{
"caption": "An energetic and anthemic Japanese rock track driven by a powerful, slightly overdriven electric guitar riff and a punchy, live-sounding drum kit. The passionate male lead vocal, delivered with a characteristic J-rock strain, soars over the driving rhythm section. The arrangement follows a classic verse-chorus structure, punctuated by a melodic and expressive guitar solo filled with bends and vibrato. A brief, more reflective bridge provides a dynamic shift before the song builds back to a powerful final chorus and a climactic outro with sustained guitar feedback.",
"lyrics": "I've removed the lyrics; you can find them in your language cover songs on YouTube. im copy from https://youtu.be/WB3hgP4q8CM?si=QC4vPkK6brhlvERH ",
"use_cot_caption": true,
"vocal_language": "ja",
"bpm": 71,
"duration": 184,
"keyscale": "A major",
"timesignature": "2",
"lm_seed": 696107280,
"inference_steps": 16,
"guidance_scale": 2,
"shift": 4,
"audio_cover_strength": 0.25,
"cover_noise_strength": 0.01,
"seed": 1499184550,
"custom_timesteps": "1->1",
"dcw_mode": "high",
"solver": "sde",
"task_type": "cover-nofsq",
"synth_model": "acestep-v15-merge-sft-turbo-xl-ta-0.7-Q8_0.gguf",
"lm_model": "acestep-5Hz-lm-4B-Q8_0.gguf",
"adapter": "cinematic-soundtrack-xl.safetensors",
"adapter_scale": 1,
"vae": "scragvae-BF16.gguf"
}
""""
link Src audio, Timbre ref : https://youtu.be/yIe2Z79FPKg?si=f5oLNCTtlMxL2Yv5 (ja sing to eng)
best case Yorushika - Sunny
""""
{
"caption": "An energetic J-rock track driven by a tight, funky rhythm section. The song opens with a clean, catchy guitar riff and a prominent, melodic bassline, establishing an upbeat groove. A powerful female lead vocal enters, delivering the verses with clarity and force. The arrangement explodes into a high-energy chorus where the guitars become more overdriven and the vocals soar with emotional intensity. The track features a dynamic structure, including a brief, melodic guitar solo and a quieter, more reflective bridge that builds back into a powerful final chorus before concluding with a stripped-down vocal and guitar phrase.",
"lyrics": "I've removed the lyrics; you can find them in your language cover songs on YouTube. im copy from https://youtu.be/zbzthAmTB50?si=8CIsGtXJ-b7OI9Qn ",
"use_cot_caption": true,
"vocal_language": "ja",
"bpm": 71,
"duration": 277,
"keyscale": "D major",
"timesignature": "3",
"lm_seed": 1936811709,
"inference_steps": 12,
"guidance_scale": 10,
"audio_cover_strength": 0.25,
"seed": 180860695,
"custom_timesteps": "1 -> 1",
"dcw_mode": "double",
"dcw_scaler": 0.01,
"dcw_high_scaler": 0.05,
"solver": "dpm3m",
"latent_shift": 0.05,
"task_type": "cover-nofsq",
"synth_model": "acestep-v15-merge-sft-turbo-xl-ta-0.7-Q8_0.gguf",
"lm_model": "acestep-5Hz-lm-4B-Q8_0.gguf",
"adapter": "cinematic-soundtrack-xl.safetensors",
"adapter_scale": 1,
"vae": "scragvae-BF16.gguf"
}
""""
link Src audio, Timbre ref : https://youtu.be/CkvWJNt77mU?si=vbjhgfbkvjqM-z8N (ja sing to eng)
I've identified the problem and am now using the URV5 model to isolate the vocals from the instrumental track. Once separated, I boost the vocal volume so that AceStep can clearly hear and process it.