Irodori-TTS
Collection
14 items • Updated • 11
Irodori-TTS-v4-Small
Irodori-TTS-v4-Small is a Japanese Text-to-Speech model based on a Rectified Flow Diffusion Transformer (RF-DiT) architecture. It combines Text (Input) + Reference Speech + Caption Text in a unified model, enabling zero-shot voice cloning, text-based voice design, and style-controlled voice cloning.
The model also supports emoji-based style and sound effect control. By inserting specific emojis into the input text, you can control speaking styles, emotions, and non-verbal vocal expressions in the generated audio.
🌟 Key Features
- Multi-modal Voice Design: Use reference audio for speaker identity and descriptive captions for voice characteristics, emotion, speaking style, and delivery.
- Long-reference Voice Cloning: Condition generation on one or more reference clips, with up to 120 seconds of combined reference audio.
- Flow Matching TTS: Rectified Flow Diffusion Transformer over continuous DACVAE latents for high-quality Japanese speech synthesis.
- Emoji-based Style Control: Embed emojis directly in the input text for granular control over the delivery and sound effects (e.g., laughter, coughing, sighs). See
EMOJI_ANNOTATIONS.mdfor details. - Integrated Watermarking: Integrates SilentCipher to apply robust, invisible audio watermarks directly to generated outputs, promoting responsible AI usage.
✨ What's New in v4
This version introduces several improvements over Irodori-TTS-600M-v3-VoiceDesign:
- Pretrained Japanese Text Encoder: Replaced the previous scratch-trained text and caption encoders with a shared, fine-tuned ModernBERT-ja backbone. This improves difficult Kanji reading and provides stronger pretrained language representations for both input text and Voice Design captions.
- Improved Text and Caption Understanding: The shared pretrained encoder strengthens linguistic understanding for text pronunciation and descriptive caption conditioning, while separate learned projectors adapt its representations to each conditioning path.
- Long Reference Support: Speaker patching and random-concatenation training extend reference conditioning to a combined maximum of 120 seconds. Multiple reference clips can be supplied and concatenated automatically.
🏗️ Architecture
The model (approximately 766M parameters) consists of five main components:
- Shared Text/Caption Encoder: A fine-tuned Japanese ModernBERT backbone shared between the input-text and Voice Design caption paths.
- Condition Projectors: Separate learned projectors map text and caption encoder representations into their respective TTS conditioning spaces.
- Reference Latent Encoder: Encodes patched reference audio latents for speaker identity conditioning, supporting up to 120 seconds of combined reference audio.
- Diffusion Transformer: Joint-attention DiT blocks combining text, reference, and caption conditioning with Low-Rank AdaLN, half-RoPE, and SwiGLU MLPs.
- Duration Predictor: Predicts audio duration from encoded text and conditioning vectors using stacked SwiGLU MLP blocks.
Audio is represented as continuous latent sequences via the Aratako/Semantic-DACVAE-Japanese-32dim codec (32-dim), enabling high-quality 48kHz waveform reconstruction.
🎧 Audio Samples
Note: To clearly demonstrate the effect of captions, the samples within each group below were generated using the exact same random seed. The variations in delivery are purely the result of the changed prompts.
1. Pure Voice Design (Text + Caption)
Generate diverse voices and styles purely through descriptive text captions without any reference audio.
| Text (Input) | Caption (Voice Design) | Generated Audio |
|---|---|---|
| 本日はお越しいただき、誠にありがとうございます。どうぞごゆっくりお過ごしください。 | 落ち着いた大人の男性。フォーマルな場で、深く響く声で丁寧かつ歓迎の意を込めて話している。 | |
| 本日はお越しいただき、誠にありがとうございます。どうぞごゆっくりお過ごしください。 | 若く元気な女性の声。カフェの店員のように、明るくハキハキとした少し高めのトーンで話している。 | |
| すみません！この近くにコンビニってありますか？ちょっと急いでて、道に迷っちゃったみたいで | 低めの声の男性が、丁寧に道を尋ねている。穏やかで礼儀正しく、余裕のある口調。 | |
| すみません！この近くにコンビニってありますか？ちょっと急いでて、道に迷っちゃったみたいで | 若い女性が、慌てた様子で早口に話している。焦りと不安が声ににじんでいる。 |
2. Style-Controlled Voice Cloning (Text + Caption + Ref Speech)
Clone a voice using reference audio, and dictate the specific emotion or delivery style using a caption.
| Text (Input) | Ref Audio | Caption (Voice Design) | Generated Audio |
|---|---|---|---|
| どうしてもっと早く教えてくれなかったの？私、ずっと待ってたのに。 | 深く傷つき、今にも泣き出しそうな様子。声が震えており、悲痛なトーンで弱々しく話す。 | ||
| どうしてもっと早く教えてくれなかったの？私、ずっと待ってたのに。 | 激しい怒りを感じており、声を荒らげている。相手を責め立てるような強い口調で、感情的なトーン。 | ||
| どうしてもっと早く教えてくれなかったの？私、ずっと待ってたのに。 | 完全に呆れ返っている様子。感情の起伏が乏しく、冷たいトーンで静かに突き放すように話す。 |
3. Fully Controlled Generation (Text + Caption + Ref Speech + Emoji)
Combine all control vectors for maximum expressiveness, adding specific physiological sounds (sighs, coughs) or distinct nuances via emojis on top of the cloned and styled voice.
| Text (with Emoji) | Ref Audio | Caption (Voice Design) | Generated Audio |
|---|---|---|---|
| あははっ🤭、それ本当に言ってるの？…😮💨まぁ、君らしいけどね。 | 余裕のある大人の男性。親しい相手に対して、くだけた雰囲気で呆れながらも楽しそうに話している。 | ||
| ゲホッ、ゲホッ🤧…ごめん、少し休ませて。😭今日はもう無理みたい。 | 体調が悪く、非常に苦しそうな若い女性。息も絶え絶えに、申し訳なさそうに弱々しい声で話している。 |
🚀 Usage
For inference code, installation instructions, and training scripts, please refer to the GitHub repository:
For lower-memory inference, torchao INT8, INT4, and FP8 variants are available at Aratako/Irodori-TTS-v4-Small-Quantized.
Long-reference Voice Cloning
For long-reference voice cloning, using multiple shorter clips from the same speaker is recommended. v4-Small was trained by randomly concatenating multiple short utterances from each speaker, and the reference-length benchmark below follows the same construction. Inference also accepts a single uninterrupted long recording, but the benefit of that input format has not been evaluated and may differ from the reported results.
📊 Benchmarks
The Irodori-TTS-v4-Small and Irodori-TTS-600M-v3-VoiceDesign results below were measured under matched inference settings. The five-seed evaluations use the consecutive base sampling seeds 0 through 4. Unless otherwise noted, values are the mean and population standard deviation across these five seeds.
The Joyo Kanji Yomi Benchmark, JSUT, Coco-Nut, and JVS datasets were not included in the training data.
Japanese Reading
Japanese reading was evaluated on the Joyo Kanji Yomi Benchmark and JSUT BASIC5000. Irodori generation used no reference audio or Voice Design caption, FP32 inference, 40 RF steps, and text CFG 3.0. The published T5Gemma-TTS and Sarashina2.2-TTS Stage 2 values are taken from the Sarashina2.2-TTS paper.
Joyo Kanji Yomi Benchmark
| Model | Kana-CER ↓ | Kana-CER clipped ↓ | Sentence Kana-CER ↓ | Standard CER ↓ |
|---|---|---|---|---|
| Irodori-TTS-600M-v3-VoiceDesign | 8.49 ± 0.21% | 5.59 ± 0.09% | 2.45 ± 0.02% | 4.88 ± 0.21% |
| Irodori-TTS-v4-Small | 7.43 ± 0.17% | 5.08 ± 0.03% | 2.89 ± 0.03% | 5.35 ± 0.18% |
| T5Gemma-TTS (paper) | 13.81 ± 2.52% | 8.55 ± 0.45% | 3.69 ± 1.85% | 5.68 ± 2.85% |
| Sarashina2.2-TTS Stage 2 (paper) | 7.83 ± 0.70% | 5.45 ± 0.10% | 3.41 ± 0.77% | 5.28 ± 0.37% |
v4-Small improved the target-kanji Kana-CER metrics in every seed. Its sentence-level Kana-CER and Standard CER were slightly worse than v3, so the improvement does not apply uniformly to every reading metric.
JSUT BASIC5000
| Model | Sentence Kana-CER ↓ | Standard CER ↓ |
|---|---|---|
| Irodori-TTS-600M-v3-VoiceDesign | 3.62 ± 0.03% | 7.19 ± 0.05% |
| Irodori-TTS-v4-Small | 3.49 ± 0.02% | 7.32 ± 0.09% |
| T5Gemma-TTS (paper) | 2.80 ± 0.04% | 7.63 ± 0.07% |
| Sarashina2.2-TTS Stage 2 (paper) | 2.91 ± 0.06% | 8.02 ± 0.07% |
Note: These published values are contextual references, not a controlled model ranking. T5Gemma-TTS and Sarashina2.2-TTS use an unpublished fixed reading-style reference speech prompt and its transcript, whereas the Irodori runs use neither reference audio nor a caption. Model architectures and inference settings also differ.
Voice Design
Voice Design prompt adherence was evaluated using all 2,890 public voice descriptions from the Coco-Nut test set. Each description was paired with the same short, medium, and long reading texts, producing 8,670 clips per model without reference audio and using one deterministic seed per description/text pair. Gemini 3.6 Flash independently scored each clip from 1 to 5 for agreement between the requested voice description and the generated speech.
| Model | Mean score ↑ | Scores 1–2 ↓ | Scores 4–5 ↑ |
|---|---|---|---|
| Irodori-TTS-600M-v3-VoiceDesign | 4.2096 | 17.09% | 73.30% |
| Irodori-TTS-v4-Small | 4.2339 | 16.78% | 74.12% |
Unlike the other benchmarks, this evaluation uses a single generation run with base seed 0. The table therefore reports aggregates over the evaluated clips rather than a mean and standard deviation across five synthesis seeds.
This is a lightweight, cost-conscious internal comparison using a single automatic judge without human validation. It is not intended as a research-grade benchmark or as standalone evidence for academic claims.
See VOICE_DESIGN_BENCHMARK.md for the complete protocol, scoring rubric, results, and limitations.
Voice Cloning and Reference Length
Voice cloning was evaluated on JVS using all 100 speakers, five target texts, and five synthesis seeds. For v4-Small, independently encoded reference utterances from each speaker were concatenated into nested one-clip, approximately 30-second, approximately 60-second, and 120-second conditions. Similarity was measured against a fixed centroid of ten held-out natural utterances per speaker, so longer conditioning references did not change the scoring target.
| Model / Reference | JVS CAM++ cosine ↑ | JVS CAM++ top-1 ↑ |
|---|---|---|
| v3 VoiceDesign, one clip | 0.6782 ± 0.0007 | 88.92 ± 0.43% |
| v4-Small, one clip | 0.6610 ± 0.0013 | 84.60 ± 0.55% |
| v4-Small, ~30 seconds | 0.7521 ± 0.0008 | 98.56 ± 0.20% |
| v4-Small, ~60 seconds | 0.7646 ± 0.0003 | 99.56 ± 0.23% |
| v4-Small, 120 seconds | 0.7753 ± 0.0009 | 99.76 ± 0.23% |
v4-Small gains substantial speaker similarity from longer references, with most of the improvement already present at approximately 30 seconds. With only one short clip, v3 achieved higher similarity. This reduction in one-clip cloning performance is considered a tradeoff of changing speaker-conditioning training from primarily single clips to random concatenation for long-reference support. ERes2Net, used as a second independently trained speaker encoder, showed the same pattern. Extending the v4 reference from one clip to 120 seconds improved CAM++ cosine by +0.1143, with a speaker-bootstrap 95% interval of [+0.1058, +0.1230].
See VOICE_CLONING_BENCHMARK.md for the dataset, reference construction, generation conditions, complete CAM++/ERes2Net/CER results, and limitations.
📊 Training Data & Annotation
The model was trained on an expanded, high-quality Japanese speech dataset. To enable the multi-modal Voice Design functionality, the training data was enriched with comprehensive text captions describing the audio characteristics.
The emoji annotations and initial text captions were generated and labeled using a fine-tuned model based on Qwen/Qwen3-Omni-30B-A3B-Instruct. Subsequently, the text captions were rephrased and refined using Qwen/Qwen3.5-35B-A3B.
⚠️ Limitations
- Japanese Only: This model currently supports Japanese text input only.
- Short-reference Voice Cloning: With only one short reference clip, objective speaker similarity was modestly lower than v3 in the evaluation. Approximately 30 seconds or more of reasonably clean reference speech is recommended when available.
- Long-reference Composition: Training and long-reference evaluation used multiple short utterances from the same speaker concatenated together. A single uninterrupted long recording is supported as input, but its effect has not been evaluated.
- Conditioning Conflicts: When using both reference audio and a text caption, contradictory instructions may result in unstable audio quality, unnatural artifacts, or one condition overriding the other. For optimal results, use the caption to guide emotion, style, or environment while keeping the base voice characteristics aligned with the reference audio.
- Prompt Adherence: While the model generally follows caption instructions, highly complex or contradictory descriptions may produce inconsistent results.
- Emoji Control: The effect of emoji-based control may vary depending on context and is not always perfectly consistent.
- Kanji Reading: Difficult Kanji reading has improved over v3, but uncommon names, specialized terminology, and context-dependent readings may still be pronounced incorrectly.
- Evaluation Scope: No large-scale human MOS, naturalness, prompt-adherence, or speaker-similarity evaluation was conducted. Automatic benchmark scores do not fully represent human perception.
📜 License & Ethical Restrictions
License
This model is released under MIT.
Ethical Restrictions
In addition to the license terms, the following ethical restrictions apply:
- No Impersonation: Do not use this model to clone or impersonate the voice of any individual (e.g., voice actors, celebrities, public figures) without their explicit consent.
- No Misinformation: Do not use this model to generate deepfakes or synthetic speech intended to mislead others or spread misinformation.
- Voice Generation Disclaimer: When generating speech purely from text or captions without using reference audio, it is possible that the generated voice may coincidentally resemble that of a real person. This is strictly a probabilistic artifact within the latent space. The model was not trained with the intent of reproducing specific individuals.
- Liability Disclaimer: The developers assume no liability for any misuse of this model. Users are solely responsible for ensuring their use of the generated content complies with applicable laws and regulations in their jurisdiction.
🙏 Acknowledgments
This project builds upon the following works:
- Echo-TTS — Architecture and training design reference
- DACVAE — Audio VAE
- sbintuitions/modernbert-ja-310m — Pretrained Japanese text and caption encoder
- SilentCipher — Audio watermarking integration
We would also like to extend our special thanks to Respair for the inspiration behind the emoji annotation feature, and to gabrielclark3330 and kikouousya for supporting this project.
🖊️ Citation
If you use Irodori-TTS in your research or project, please cite it as follows:
@misc{irodori-tts-v4-small,
author = {Chihiro Arata},
title = {Irodori-TTS: A Flow Matching-based Text-to-Speech Model with Emoji-driven Style Control},
year = {2026},
publisher = {Hugging Face},
journal = {Hugging Face repository},
howpublished = {\url{https://huggingface.co/Aratako/Irodori-TTS-v4-Small}}
}