anuj1541/cidco-marathi-english-qa-scored
Viewer • Updated • 5.97k • 19
How to use anuj1541/indictrans2-en-indic-mr-cidco with Transformers:
# Use a pipeline as a high-level helper
# Warning: Pipeline type "translation" is no longer supported in transformers v5.
# You must load the model directly (see below) or downgrade to v4.x with:
# pip install "transformers<5.0.0"
from transformers import pipeline
pipe = pipeline("translation", model="anuj1541/indictrans2-en-indic-mr-cidco", trust_remote_code=True) # pip install -U transformers accelerate
# Load model directly
from transformers import AutoModelForSeq2SeqLM
model = AutoModelForSeq2SeqLM.from_pretrained("anuj1541/indictrans2-en-indic-mr-cidco", trust_remote_code=True, device_map="auto")Fine-tuned from ai4bharat/indictrans2-en-indic-dist-200M on the combined anuj1541/cidco-marathi-english-qa-scored and anuj1541/marathi-english-cidco-long-docs datasets.
Training mode: LoRA merged into the base (r=16, lr=0.0001).
from transformers import AutoModelForSeq2SeqLM, AutoTokenizer
from IndicTransToolkit import IndicProcessor
repo = "anuj1541/indictrans2-en-indic-mr-cidco"
tokenizer = AutoTokenizer.from_pretrained(repo, trust_remote_code=True)
model = AutoModelForSeq2SeqLM.from_pretrained(repo, trust_remote_code=True)
eng_Latn -> mar_Devaconfiguration_indictrans.py, modeling_indictrans.py, tokenization_indictrans.py, dict.SRC.json, dict.TGT.json, model.SRC, model.TGT, pytorch_model.bin, config.json, tokenizer_config.json, generation_config.json
| Model | BLEU | ChrF | METEOR | TER | COMET | BERTF1 | Judge/5 |
|---|---|---|---|---|---|---|---|
base (ai4bharat/indictrans2-en-indic-dist-200M) |
35.87 | 56.52 | 37.17 | 203.38 | 0.6741 | 81.22 | 2.47 |
| fine-tuned (this repo) | 57.55 | 66.23 | 48.98 | 106.01 | 0.6917 | 84.34 | 2.51 |
Base model
ai4bharat/indictrans2-en-indic-dist-200M