rmacario/nhengatu-xlmr

XLM-R fine-tuned for Portuguese–Nheengatu sentence retrieval

Fine-tuned XLM-RoBERTa base for cross-lingual sentence retrieval between Portuguese (pt) and Nheengatu (yrl), a Tupian language of the Rio Negro region in Brazil.

The model maps parallel Portuguese and Nheengatu sentences to nearby points in a shared embedding space, so a Portuguese query can retrieve its Nheengatu translation (and vice versa).

Training

• Base model: xlm-roberta-base
• Objective: Multiple Negatives Ranking Loss (contrastive)
• Data: 4,997 Portuguese–Nheengatu sentence pairs (Constitution, Tycho Brahe fragments, and grammars via the CompLin/nheengatu resources)
• Split: 80/20 train/test (seed 42)
• Hyperparameters: batch size 16, learning rate 2e-5, 3 epochs, max length 128
• Pooling: mean pooling over the last hidden state

Evaluation

Cross-lingual sentence retrieval on the 1,000-pair held-out test set:

Metric	Score
P@1	0.247
P@5	0.504
P@10	0.621
MRR	0.371

Usage

from transformers import AutoTokenizer, AutoModel
import torch, torch.nn.functional as F

tok = AutoTokenizer.from_pretrained("rmacario/nhengatu-xlmr")
model = AutoModel.from_pretrained("rmacario/nhengatu-xlmr").eval()

def embed(texts):
    enc = tok(texts, padding=True, truncation=True, max_length=128, return_tensors="pt")
    with torch.no_grad():
        out = model(**enc)
    mask = enc["attention_mask"].unsqueeze(-1).float()
    emb = (out.last_hidden_state * mask).sum(1) / mask.sum(1)
    return F.normalize(emb, p=2, dim=1)

pt = embed(["Existe muita água no rio."])
yrl = embed(["Aikué siía í paraná upé."])
print((pt @ yrl.T).item())  # cosine similarity

Limitations

Trained on written, formal registers only, and on a small corpus. Retrieval accuracy is modest (P@1 ≈ 0.25) and the model is intended for research on low-resource cross-lingual transfer, not production use. Any deployment for Nheengatu language technology should involve the language community.

Data and code

• Corpus and experiments: https://huggingface.co/rmacario/nhengatu-experiments
• Code: https://github.com/rmaacario/nhengatu-constitution