nllb200-formosan-en-spm8k

Direction: Formosan to English
Base model: facebook/nllb-200-distilled-600M
Recipe: nllb200-spm8k-directional-v9
Release: 20260828-121226, validation-selected step 290,000

This is a directional model for 15 Formosan languages. It uses the private_no_bible leakage-controlled corpus, Formosan-aware 8k SentencePiece extension, and explicit direction, source-language, and dialect controls. It does not infer or encode domains from repository or path names. Training uses language-temperature sampling with alpha 0.5; sentence rows have weight 1.0 and explicit lexical rows, when present, have weight 0.25. Dialect tags use 25% dropout so default is a normal inference condition. Synthetic pivot rows are train-only. The model weights are public. The training corpus is distributed separately to authorized FormosanBank members through the access-controlled FormosanBank/formosan-mt-private dataset and is not included with the weights.

Model details

Item	Value
Base revision	f8d333a098d19b4fd9a8b18f94170487ad3f821d
Corpus release	20260824T042425Z_v3
Training code commit	a2cfed01277bc30bd9dea6a96af26564bb4f0718
Total corpus rows	601,091
Training rows	539,213
Human split policy	70% train / 10% validate / 20% test
Synthetic evaluation policy	train_only_after_human_split
Language sampling alpha	0.5
Sentence / lexical sampling weight	1.0 / 0.25
Dialect-tag dropout	25%
Training budget	300,000 updates
Effective batch size	64
Maximum sequence length	384
Learning rate	2e-05
Precision	bf16
Checkpoint selection	Validation chrF2
Formosan text	kindOf=standard, formosan-mt-standard-v3
Corpus SHA-256	e5948794814525d53a3bf566b0b6f5c0b58c8bfdf55e909f56c7eb24ebc41e98
Training profile SHA-256	01637c79ae86f8cc9c149a61b4a24e808d18261af140afe4fb0508a5190ce62a

Usage

import torch
from transformers import AutoModelForSeq2SeqLM, NllbTokenizer
from formosan_mt_inference import normalize_formosan


model_id = "FormosanBank/nllb200-formosan-en-spm8k"
tokenizer = NllbTokenizer.from_pretrained(model_id, use_fast=False)
model = AutoModelForSeq2SeqLM.from_pretrained(model_id)
model.to("cuda" if torch.cuda.is_available() else "cpu")
NLLB_LIDS = {'ami': 'ami_Latn', 'bnn': 'bnn_Latn', 'ckv': 'ckv_Latn', 'dru': 'dru_Latn', 'pwn': 'pwn_Latn', 'pyu': 'pyu_Latn', 'ssf': 'ssf_Latn', 'sxr': 'sxr_Latn', 'szy': 'szy_Latn', 'tao': 'tao_Latn', 'tay': 'tay_Latn', 'trv': 'trv_Latn', 'tsu': 'tsu_Latn', 'xnb': 'xnb_Latn', 'xsy': 'xsy_Latn'}

def translate(text, lang_code, dialect="default"):
    text = normalize_formosan(text, lang_code)
    tokenizer.src_lang = NLLB_LIDS[lang_code]
    dialect_tag = f"<dialect_{dialect}>"
    if tokenizer.convert_tokens_to_ids(dialect_tag) == tokenizer.unk_token_id:
        dialect_tag = "<dialect_default>"
    prompt = f"<to_eng> <src_{lang_code}> {dialect_tag} {text}"
    inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
    output = model.generate(
        **inputs,
        decoder_start_token_id=tokenizer.eos_token_id,
        forced_bos_token_id=tokenizer.convert_tokens_to_ids('eng_Latn'),
        max_new_tokens=256,
        num_beams=4,
    )
    return tokenizer.batch_decode(output, skip_special_tokens=True)[0]

print(translate("Pa'araw cingra.", "ami"))

The direction, source-language, and dialect tags are part of the training contract. Use default when dialect metadata is unavailable. This model does not use domain or repository tags.

Evaluation

The best checkpoint was selected on validation chrF2. Test and validation contain only eligible, human-translated sentence pairs. Synthetic pivots and lexical entries are train-only. The headline result uses default metadata controls (the default dialect only), so it does not assume access to test-set dialect labels or any inferred domain metadata.

Split	Rows
Train	539,213
Test	41,248
Validate	20,630

Scope	BLEU	chrF2	TER
Hard test	21.29	38.85	69.25
Selection validation	12.75	33.01	78.68

Test empty-output rate: 0.0000%.

Bootstrap confidence intervals were not requested for this release.

Metric signatures:

• BLEU: nrefs:1|case:mixed|eff:yes|tok:13a|smooth:exp|version:2.5.1
• chrF2: nrefs:1|case:mixed|eff:yes|nc:6|nw:0|space:no|version:2.5.1
• TER: nrefs:1|case:mixed|tok:tercom|norm:yes|punct:yes|asian:no|version:2.5.1

Language	Samples	BLEU	chrF2	TER
ami	4,223	17.46	36.71	72.51
bnn	5,721	24.94	41.72	67.24
ckv	2,926	16.44	35.38	69.40
dru	4,624	19.00	34.86	73.12
pwn	5,486	22.56	40.78	70.83
pyu	4,145	20.36	38.68	65.32
ssf	1,075	31.98	48.33	57.56
sxr	992	19.22	37.26	70.90
szy	639	27.05	44.49	63.60
tao	681	25.62	42.84	65.02
tay	4,129	23.94	40.20	67.54
trv	2,786	21.08	39.18	70.96
tsu	986	21.23	38.93	67.60
xnb	1,744	15.63	35.47	71.95
xsy	1,091	25.43	44.11	63.63

The corpus gate applies 70% train / 10% validate / 20% test to deduplicated human pairs within each language and source where capacity permits. Synthetic pivots are appended to training after that split. Evaluation is sentence-only and contains no lexical or synthetic rows. Standard-tier Formosan text is normalized with formosan-mt-standard-v3. The release requires zero exact, skeleton, one-edit, or configured high character n-gram train/evaluation conflicts. Document overlap is diagnostic. This release passed all leakage gates: exact 0, skeleton 0, one-edit 0, character n-gram 0. Document overlap: 925.

See eval/metrics.json for sacreBLEU signatures, per-language, source-corpus, dialect, and length diagnostics. publication.json records the corpus, profile, run, and checkpoint hashes used for this release.

Intended use

This model supports research, corpus development, and assisted translation for the 15 included Formosan languages. It is designed for the exact prompt and generation contract shown above.

Limitations

Outputs require knowledgeable speaker review. Aggregate metrics hide large differences among languages and source collections. This model is not suitable for authoritative, medical, legal, or safety-critical translation.