ILRDF/nllb-600m-formosan-all-finetune-v3

NLLB-600M Formosan (all dialects) finetune v3

A finetune of facebook/nllb-200-distilled-600M for translation between Mandarin (Traditional Chinese) and 42 Formosan dialects of Taiwan, in both directions. It succeeds ILRDF/nllb-600m-formosan-all-finetune-v2.

Changes from v2:

• New training data (10 datasets), with every pair that matches the test set removed and duplicates dropped.
• Byte-fallback tokenizer: characters outside NLLB's vocabulary (蕉, 猴, 橘, ...) are no longer lost as <unk>.
• Sentence-initial casing normalized for Amis, Bunun, Truku and Seediq.
• Klokah lessons merged into paragraphs, so the model also sees multi-sentence input.
• Synthetic repeated-word pairs (fafoy, fafoy, fafoy ↔ 豬、豬、豬), which v2 collapses to one word.
• Larger batches (256 sentences/step), higher learning rate and 10 epochs.

Files

File	Description
model.safetensors, config.json, generation_config.json	Model weights
tokenizer.json, tokenizer_config.json	NLLB tokenizer + 256 byte-fallback tokens + 42 dialect tokens (vocabulary 256,509)
dialects.json	lang_code → dialect token, dialect name
formosan_translate.py	Inference (FormosanTranslator)

Usage

pip install torch transformers tokenizers huggingface_hub

import sys

from huggingface_hub import snapshot_download

repo_dir = snapshot_download("ILRDF/nllb-600m-formosan-all-finetune-v3")
sys.path.insert(0, repo_dir)
from formosan_translate import FormosanTranslator

mt = FormosanTranslator(repo_dir, device="cuda")
print(mt.translate(["Maolah kako komaen to pawli."], "ami-x-pswl", "zh"))  # 海岸阿美語 → 華語
print(mt.translate(["我喜歡吃香蕉。"], "zh", "ami-x-pswl"))               # 華語 → 海岸阿美語

• Languages are formospeech lang_codes (table below) or "zh"; the model tokens (ami_Coas, zho_Hant) also work.
• translate() takes a list and batches it (batch_size=32); decoding is greedy by default (num_beams=1), as in the evaluation.
• Load the tokenizer as formosan_translate.py does. AutoTokenizer.from_pretrained() on this repo silently drops byte fallback (transformers 5.x rebuilds the tokenizer without it), and every character outside NLLB's vocabulary becomes <unk>.
• Amis, Bunun, Truku and Seediq sentence starts are lower-cased in training; translate() lower-cases them in the input and capitalizes them in the output. Case is never changed in Kavalan and Saisiyat, where it marks different sounds.

42 dialects and their tokens

lang_code	Dialect	Token
ami-x-frng	馬蘭阿美語	ami_Mala
ami-x-iams	南勢阿美語	ami_Sout
ami-x-pld	恆春阿美語	ami_Heng
ami-x-pswl	海岸阿美語	ami_Coas
ami-x-skl	秀姑巒阿美語	ami_Xiug
bnn-x-bkh	卡群布農語	bnn_Kaqu
bnn-x-bnz	巒群布農語	bnn_Luan
bnn-x-isbk	郡群布農語	bnn_Junq
bnn-x-td	卓群布農語	bnn_Zhuo
bnn-x-vtn	丹群布農語	bnn_Tanq
ckv	噶瑪蘭語	ckv_Kava
dru-x-kgdv	多納魯凱語	dru_Dona
dru-x-lbw	大武魯凱語	dru_Dawu
dru-x-ngdr	霧臺魯凱語	dru_Wuta
dru-x-opnh	萬山魯凱語	dru_Wans
dru-x-tldr	茂林魯凱語	dru_Maol
dru-x-trmk	東魯凱語	dru_East
pwn-x-kcdsn	東排灣語	pwn_East
pwn-x-pnvn	中排灣語	pwn_Cent
pwn-x-vnrn	北排灣語	pwn_Nrth
pwn-x-ynvl	南排灣語	pwn_Sout
pyu-x-ksvk	建和卑南語	pyu_Jian
pyu-x-ktrp	知本卑南語	pyu_Zhib
pyu-x-mkzy	西群卑南語	pyu_Xiqu
pyu-x-pym	南王卑南語	pyu_Nanw
ssf	邵語	ssf_Thao
sxr	拉阿魯哇語	sxr_Saar
szy	撒奇萊雅語	szy_Saki
tao	雅美語	tao_Yami
tay-x-cql	四季泰雅語	tay_Four
tay-x-kls	宜蘭澤敖利泰雅語	tay_Yzea
tay-x-mtuw	汶水泰雅語	tay_Wens
tay-x-plngw	萬大泰雅語	tay_Wand
tay-x-sql	賽考利克泰雅語	tay_Seko
tay-x-sul	澤敖利泰雅語	tay_Zeao
trv-x-td	都達賽德克語	trv_Duda
trv-x-tgdy	德固達雅賽德克語	trv_Tegu
trv-x-trk	德鹿谷賽德克語	trv_Delu
trv-x-truku	太魯閣語	trv_Truk
tsu	鄒語	tsu_Tsou
xnb	卡那卡那富語	xnb_Kana
xsy	賽夏語	xsy_Sais

Training

Data

train splits of ten datasets. Every pair is used in both directions (Formosan → Mandarin and Mandarin → Formosan). Records below are both directions, before deduplication.

Dataset	Revision	Records
formospeech/klokah	85d5ac7e	599,522
formospeech/ilrdf_dicts	51802fc7	180,254
formospeech/formosan_bible	d682be95	178,678
ILRDF/formosan_db	b0faf438	154,084
formospeech/nchu_pangcah	10885872	60,884
formospeech/ithuan_formosan_text	6b1de5e0	41,110
formospeech/nchc_formosan	3d92966b	34,212
formospeech/ithuan_formosan	ec3c2cba	31,494
formospeech/ntu_formosan_corpus	26245016	27,978
formospeech/klokah_pcc	c0bc98e3	18,662
Synthetic repeated-word pairs	—	12,600

After removing 3,944 records that match the test set and 132,628 duplicates: 1,202,906 records (601,453 pairs).

Processing:

1. Dropped: rows whose Formosan or Mandarin side is empty or punctuation only, glossary entries (ids containing word), and dialects outside the 42.
2. Leading punctuation removed; a trailing , / : becomes ., a trailing ，、： becomes 。.
3. Sentence starts lower-cased for Amis, Bunun, Truku and Seediq, whose orthographies capitalize them but whose ASR-derived transcripts are mostly lower case.
4. Klokah reading lessons (cu_practice, rd_practice) are merged into paragraphs ending on a Chinese full stop and used only as paragraphs. 10% of the texts go to the paragraph test set, split by text across all its dialect versions.
5. Repeated-word pairs: 150 klokah glossary words per dialect, repeated 2–6 times, separated by , or a space.
6. Test-set exclusion: pairs are matched on (dialect, Formosan, Mandarin) with punctuation and whitespace removed and case folded (except Kavalan and Saisiyat). Every match of a test sentence, a test paragraph, or a sentence inside a test paragraph is removed.
7. Deduplication on the same key, across all datasets, keeping the first occurrence.

Tokenizer

The NLLB tokenizer with new tokens appended after its vocabulary, so every original token id is unchanged:

• 256 byte tokens <0x00>–<0xFF> with byte fallback on. 0.32% of the characters in the training data, in 23% of the Mandarin sentences, are outside NLLB's vocabulary.
• 42 dialect tokens (ami_Coas, pwn_Cent, ...), one per dialect.

Configuration

• 4× RTX A5000, 10 epochs (46,990 steps), 256 sentences per step (64 per GPU), lr 1.5e-3 (linear, 10% warmup), weight decay 0.01, AdamW, bf16, max input length 512 tokens.
• Batches group sentences of similar length and are balanced across the GPUs by estimated compute.

Evaluation

• Sentence test set: the eval splits of formospeech/klokah and formospeech/ithuan_formosan, 15,450 sentences in each direction (not used in training).
• Paragraph test set: 1,270 klokah paragraphs in each direction (see Processing, step 4).
• Metrics: sacrebleu corpus chrF, chrF++ (word n-gram order 2) and BLEU, case-insensitive except for Kavalan and Saisiyat targets. BLEU tokenizes Mandarin with zh (one token per character) and Formosan with 13a. For Mandarin targets, punctuation on both sides is converted to full width first, and chrF++ is not reported: its word n-grams split on whitespace, which unsegmented Mandarin does not have.
• Greedy decoding; Amis, Bunun, Truku and Seediq sentence starts handled as in Usage.
• Baseline: ILRDF/nllb-600m-formosan-all-finetune-v2. Its training data contains 13,978 of the 15,450 test sentences, so the comparison that matters is the 1,472 it has not seen (v2-unseen).

Test set	Direction	v2 chrF	v2 chrF++	v2 BLEU	v3 chrF	v3 chrF++	v3 BLEU
Sentences (15,450)	Formosan → Mandarin	76.19	—	78.90	83.94	—	85.85
Sentences (15,450)	Mandarin → Formosan	68.41	67.10	48.10	67.27	65.72	45.37
v2-unseen (1,472)	Formosan → Mandarin	42.72	—	45.84	81.97	—	83.31
v2-unseen (1,472)	Mandarin → Formosan	47.46	45.49	22.13	55.84	54.04	30.83
Paragraphs (1,270)	Formosan → Mandarin	9.69	—	8.66	12.85	—	13.14
Paragraphs (1,270)	Mandarin → Formosan	33.61	31.27	5.28	39.40	36.65	9.54

• v2 is ahead only on Mandarin → Formosan over all sentences, most of which it was trained on.
• Much of the Formosan → Mandarin gain on v2-unseen comes from other dialect versions of the same klokah sentences, which are in v3's training data: 1,206 of the 1,472 Mandarin sentences appear there, nearly all paired with another dialect.
• Byte fallback is worth about 4.7 chrF on Formosan → Mandarin over all sentences: 16% of the references contain a character outside NLLB's vocabulary, and v3 produces 89% of those characters correctly.

Per-dialect results (v3)

→ zh: Formosan → Mandarin; zh →: Mandarin → Formosan; para: paragraph test set.

lang_code	Dialect	n	→ zh chrF	→ zh BLEU	zh → chrF	zh → chrF++	zh → BLEU	n (para)	para → zh chrF	para zh → chrF	para zh → chrF++
ami-x-frng	馬蘭阿美語	349	87.42	89.17	76.17	74.88	57.77	40	18.35	41.62	40.19
ami-x-iams	南勢阿美語	316	86.41	87.82	69.03	67.65	48.29	36	18.50	42.64	40.63
ami-x-pld	恆春阿美語	284	88.11	89.70	71.68	69.63	48.54	33	13.79	33.94	32.47
ami-x-pswl	海岸阿美語	447	91.88	92.50	69.09	67.39	45.42	22	10.24	38.85	36.37
ami-x-skl	秀姑巒阿美語	317	75.97	78.32	67.48	66.01	45.54	26	10.25	37.20	35.55
bnn-x-bkh	卡群布農語	298	83.72	85.03	59.55	56.85	30.95	19	7.85	31.90	28.94
bnn-x-bnz	巒群布農語	304	82.47	84.47	65.65	63.23	39.97	32	13.83	42.00	38.87
bnn-x-isbk	郡群布農語	575	83.62	85.13	66.50	63.96	39.46	30	14.97	35.45	32.25
bnn-x-td	卓群布農語	320	85.83	86.41	66.25	64.16	41.90	29	11.33	36.26	32.49
bnn-x-vtn	丹群布農語	307	77.91	81.12	64.97	62.76	39.27	13	9.70	39.13	36.21
ckv	噶瑪蘭語	459	88.18	89.84	72.77	71.92	53.65	25	9.79	36.01	33.87
dru-x-kgdv	多納魯凱語	262	79.99	80.30	63.99	60.93	35.05	21	7.28	34.62	29.92
dru-x-lbw	大武魯凱語	314	82.78	84.59	69.09	67.21	45.51	46	12.04	43.00	39.99
dru-x-ngdr	霧臺魯凱語	452	87.24	88.37	62.44	59.56	32.52	27	8.63	33.52	28.94
dru-x-opnh	萬山魯凱語	292	75.05	77.56	64.35	61.06	33.83	24	9.94	31.06	27.38
dru-x-tldr	茂林魯凱語	254	70.79	69.99	50.92	48.91	29.01	21	3.79	23.58	21.08
dru-x-trmk	東魯凱語	291	83.80	85.28	63.99	60.69	33.97	40	9.73	33.79	30.06
pwn-x-kcdsn	東排灣語	295	80.18	82.89	66.36	64.53	43.16	34	10.10	35.11	31.69
pwn-x-pnvn	中排灣語	298	77.41	79.69	67.95	66.36	48.07	52	7.94	32.21	29.03
pwn-x-vnrn	北排灣語	567	80.76	82.99	67.41	65.87	46.22	78	7.22	35.70	32.23
pwn-x-ynvl	南排灣語	295	84.05	85.64	66.03	63.82	41.30	30	8.33	34.99	31.89
pyu-x-ksvk	建和卑南語	280	89.98	91.85	74.75	73.97	56.77	37	21.81	49.24	46.40
pyu-x-ktrp	知本卑南語	295	90.28	91.57	72.15	71.31	54.18	62	24.09	47.23	44.43
pyu-x-mkzy	西群卑南語	299	77.04	83.21	73.39	72.09	50.80	60	26.70	56.35	53.77
pyu-x-pym	南王卑南語	471	81.25	82.47	70.22	69.35	52.68	32	18.94	48.29	46.10
ssf	邵語	373	82.86	84.27	69.17	67.70	46.25	12	11.80	42.16	40.56
sxr	拉阿魯哇語	442	85.00	86.89	72.88	68.71	37.95	34	6.27	38.56	32.78
szy	撒奇萊雅語	520	85.18	86.61	70.38	69.27	50.42	9	10.20	37.93	35.57
tao	雅美語	450	74.18	76.02	56.82	55.82	34.70	40	6.88	32.48	30.69
tay-x-cql	四季泰雅語	298	86.62	88.29	62.07	61.31	41.37	30	10.89	30.42	28.12
tay-x-kls	宜蘭澤敖利泰雅語	315	90.15	90.94	69.09	68.60	52.11	5	12.11	37.92	35.32
tay-x-mtuw	汶水泰雅語	288	85.34	86.84	65.47	64.11	46.54	9	6.40	40.17	36.88
tay-x-plngw	萬大泰雅語	292	86.90	88.36	67.46	66.98	51.58	27	9.17	34.61	32.96
tay-x-sql	賽考利克泰雅語	548	87.38	88.51	64.06	62.91	44.77	15	12.39	32.38	30.33
tay-x-sul	澤敖利泰雅語	268	89.35	90.75	63.66	62.90	44.08	31	12.51	36.72	34.61
trv-x-td	都達賽德克語	288	90.28	91.87	67.45	67.14	48.45	10	13.53	42.30	40.25
trv-x-tgdy	德固達雅賽德克語	488	80.16	82.38	61.00	60.46	41.29	16	10.78	38.13	36.51
trv-x-trk	德鹿谷賽德克語	295	85.35	87.44	71.88	71.13	54.07	4	10.28	41.21	38.64
trv-x-truku	太魯閣語	482	86.13	87.83	69.68	69.21	51.90	45	16.45	44.36	43.03
tsu	鄒語	532	81.87	82.40	62.89	61.28	40.04	21	10.34	34.79	31.85
xnb	卡那卡那富語	479	83.91	85.96	70.63	68.67	44.36	53	15.96	45.32	41.87
xsy	賽夏語	451	90.67	91.39	71.96	70.55	53.80	40	11.63	47.18	44.36

Limitations

• Paragraph translation is much weaker than sentence translation, especially into Mandarin (chrF 12.85). Translate long text sentence by sentence.
• Mandarin → Formosan is harder than the other direction (sentence chrF 67.27 vs 83.94), and varies widely by dialect; see the per-dialect table.
• The test sets come from the same sources as the training data (klokah lessons, ithuan_formosan), so scores on other kinds of text (news, conversation) are likely lower.
• Use the tokenizer loading in formosan_translate.py; a plain AutoTokenizer.from_pretrained() loses byte fallback.