kcsheang/khmerclipper-trocr-khmer

KhmerClipper TrOCR (Khmer + English)

A TrOCR model fine-tuned to read single text lines in Khmer, English and mixed Khmer/English. It is the model behind KhmerClipper, an on-device screen OCR app.

These are the full PyTorch weights (safetensors), ready for inference or further fine-tuning. The Windows/Linux app uses the ONNX export at kcsheang/khmerclipper-trocr-khmer-onnx.

Usage

from transformers import TrOCRProcessor, VisionEncoderDecoderModel
from PIL import Image

repo = "kcsheang/khmerclipper-trocr-khmer"
processor = TrOCRProcessor.from_pretrained(repo)
model = VisionEncoderDecoderModel.from_pretrained(repo).eval()

image = Image.open("line.png").convert("RGB")  # one line of text
pixel_values = processor(images=image, return_tensors="pt").pixel_values
ids = model.generate(pixel_values, max_new_tokens=128)
print(processor.batch_decode(ids, skip_special_tokens=True)[0])

The model reads one line at a time. For a paragraph or a screenshot, split it into lines first. KhmerClipper's line splitter is in the app source.

Fine-tuning

The checkpoint loads into the standard VisionEncoderDecoderModel / Seq2SeqTrainer workflow. The training scripts and the exact recipes for every version are in the repo: training/trocr and docs/PROJECT.md.

• Tokenizer: the byte-level BPE tokenizer from the base model, plus one added token per Khmer character (U+1780–17FF, U+19E0–19FF and the zero-width separators). Each Khmer character is one token instead of three bytes. Keep this tokenizer when you fine-tune.
• Labels were capped at 128 tokens.
• Each version was fine-tuned from the previous one at a lower learning rate (v1: 2e-5, 3 epochs; v2: 1e-5, 1 epoch; M2: 5e-6, 1 epoch from v3). If you add your own data, mix in some of the original sources so the model doesn't forget them.

Results

Character error rate (CER, lower is better) on a frozen set of 200 line images. Numbers are on-device CoreML runs on Apple Silicon, including the app's Latin-line fallback.

Engine	All	printed	ui	mixed
Tesseract 5 khm+eng (stock)	0.2580	0.5370	0.0199	0.0894
TrOCR before fine-tuning on new data (v0)	0.0593	0.0433	0.0662	0.0950
v3 (model M1)	0.0035	0.0036	0.0030	0.0046
This model (M2)	0.0034	0.0033	0.0030	0.0046

On Chrome-rendered screen captures of 1–5 lines in Apple and Khmer OS fonts, CER is 0.016 (v3: 0.016; 3.3% fewer character errors). On real handwriting test lines read through the app, M2 makes about 6% fewer errors than v3; the model alone on clean crops, 24% fewer (CER 0.0071 to 0.0054).

Earlier checkpoints saw the SoyVitou test split, which the printed fixtures come from, so the printed number is optimistic. The ui and mixed subsets were never in any training data. Full write-ups are in training/eval/results.

Training data

Public Hugging Face line-image datasets (listed in the metadata above), plus sets rendered locally with OFL Google Fonts and, from M2, the Khmer OS family (LGPL-2.1+):

• synth: 400k multi-font Khmer and mixed lines
• synth_en: 150k clean English lines, with text from English Wikipedia (CC BY-SA 4.0)
• synth_screen: 204k small screen-text strips at 11–24 px
• M2 renders: 100k general lines, 100k screen strips and 60k handwriting-font strips that add the Khmer OS faces

Every evaluation string was removed from the training text. The datasets come with their own licenses: some are MIT or CC-BY-4.0, and a few don't state one. See the dataset table for details. Thanks to all the dataset authors.

Training ran on a single RTX 3090 (bf16, Seq2SeqTrainer). This release is M2, from training run v7b (its final step). Model release numbers (M1 = v3, M2 = v7b) are separate from experiment names.

Limitations

• Single text lines only, with no layout analysis.
• Strongest on printed and screen text. Handwriting is supported, but less tested. Very small marks (one or two characters) are often misread.
• Lines much wider than about 12× their height should be decoded in pieces.