OTA-SBERTurk-Court (ota-sbert-kadisicilleri)

OTA-SBERTurk-Court is a sentence-embedding model for Latin-transcribed Ottoman Turkish legal text. We started from BERTurk and adapted it to the language of the Istanbul court registers (kadı sicilleri, 1513–1911) in two steps: first unsupervised SimCSE, then training on pairs of case summaries and records with Multiple Negatives Ranking Loss.

The model turns a sentence or short passage into a 768-dimensional vector. Cosine similarity between vectors can be used for semantic search, clustering, and scoring texts against psychological constructs.

Base model dbmdz/bert-base-turkish-cased (111M parameters)
Output dimension 768
Max sequence length 256 tokens (longer inputs are truncated)
Pooling mean
Similarity cosine
Language Ottoman Turkish, Latin transcription

Usage

from sentence_transformers import SentenceTransformer, util

model = SentenceTransformer("enesyila/ota-sbert-kadisicilleri")

query = "Zeyd, Amr'dan on bin akçe talep eyledi."
records = [
    "Mahalleden birisi mahkemeye gelüb komşusundan borcu olan bir miktar parayı istedi.",
    "Mezbûr Hind, vakf olunan menzilin tevliyetine nâib nasb olundu.",
]

q = model.encode(query, normalize_embeddings=True)
r = model.encode(records, normalize_embeddings=True)
print(q.shape)                      # (768,)
print(util.cos_sim(q, r))           # higher = more similar -- the output for the example above is tensor([[0.2620, 0.0773]]).

You don't need an instruction prefix. Inputs longer than 256 tokens are cut off, so for long records it is better to split them into sentences or passages and pool the scores.

Training

We trained the model on the Istanbul Qadi Registers corpus (İSAM; 54,670 records from 100 volumes and 21 courts). Every record comes with a short, one-sentence case summary.

Data

  1. Cleaning. We dropped records with no text or no summary, texts shorter than 20 characters or with less than 30% alphabetic characters, texts that were only short markup tags, and summaries of 5 characters or fewer. Exact duplicate (summary, text) pairs were also removed. This left 54,535 records.
  2. An additional test set for expert annotation. The 440 records annotated by the three experts, plus any exact duplicates of them, were kept out of training, validation, and checkpoint selection. The evaluation done through this test set was not covered in this model card.
  3. Split by summary. The remaining 54,095 records were split 80/10/10 by case summary (seed 42), so a summary never appears in more than one split. This gives 43,108 training, 5,715 validation, and 5,272 test records.

A note on transcription: the texts do not follow the IJMES or İslâm Ansiklopedisi transcription rules. For example, kaf and kef are not distinguished and both are written as k.

Stage 1: SimCSE (unsupervised)

Annotated data is scarce in Ottoman Turkish NLP, so the first stage uses no labels at all. Starting from dbmdz/bert-base-turkish-cased, each training record is paired with itself. Dropout (0.1) makes the two encodings slightly different, and the other records in the batch serve as negatives (SimCSE; Gao et al., 2021). This teaches the encoder the vocabulary and spelling of the registers.

Setting Value
Data 43,108 training records (record text only)
Loss CachedMultipleNegativesRankingLoss (scale 20, cosine similarity)
Batch size 128 (gradient caching, mini-batches of 32)
Epochs / learning rate 1 / 1e-5, linear schedule, warm-up ratio 0.1
Max sequence length 256 tokens
Precision bf16 mixed precision
Checkpoint selection lowest validation loss (checked every 100 steps)

Stage 2: summary and record pairs (supervised)

In the second stage the Stage 1 model learns to place a case summary close to its own record. The summary is the anchor, its record is the positive, and the other records in the batch are negatives. In this way the model learns to match short modern-Turkish descriptions with long Ottoman texts.

Setting Value
Data 43,108 (summary, record) pairs
Loss CachedMultipleNegativesRankingLoss (scale 20, cosine similarity)
Batch size 64 in-batch negatives (mini-batches of 16 only save memory)
Epochs / learning rate 2 / 2e-5, linear schedule, warm-up ratio 0.1, weight decay 0.01
Max sequence length 256 tokens (the first 256 tokens of each record)
Precision fp32 weights with bf16 mixed precision
Checkpoint selection best validation NDCG@10 (summary-to-record retrieval on 10,000 validation records, checked every 150 steps)
Batching no_duplicates sampler, so a batch never contains the same text twice
Seed 42

Neither stage uses an instruction prefix. The released model is the Stage 2 checkpoint. Training used Sentence Transformers 6.2.0, Transformers 5.20.1, and PyTorch 2.11.

Evaluation

We evaluated the model on four tasks that each check something different: whether it can find the right record from a summary (retrieval), whether matched summaries and records sit close together in the embedding space, whether it can tell related from unrelated Ottoman sentences (STS), and whether it keeps psychological constructs apart (QIC). As an external baseline we also report SerBERT, a publicly available Ottoman Turkish BERT-based MLM encoder, on the STS and QIC tasks.

1. Document retrieval (summary → record)

Each case summary is a query, and the model ranks all 5,272 test records by cosine similarity. A record counts as relevant if it belongs to the query summary. When several records share a summary, all of them are relevant. Every model gets the same queries, candidates, and relevance judgments. Recall@k is the share of a summary's records found in the top k, MRR@10 reflects the rank of the first relevant record, and NDCG@10 scores the ranking of all relevant records in the top ten.

Model NDCG@10 MRR@10 R@1 R@5 R@10
BERTurk (base) 0.011 0.009 0.006 0.013 0.019
+ SimCSE (stage 1) 0.024 0.018 0.011 0.029 0.041
OTA-SBERTurk-Court (stage 2) 0.898 0.883 0.843 0.925 0.946

The base model is close to zero because it was never trained to connect short modern-Turkish summaries with Ottoman records. The big jump comes from the summary–record training in stage 2, not from SimCSE alone.

2. Embedding space

On the test split, we checked whether matched texts sit closer together than unmatched ones, and report ROC-AUC for two checks:

  • Summary → text: is a summary more similar to its own record than to a record belonging to another summary?
  • Cross-document: are two records that share a summary more similar to each other than to a record with a different summary? Records with the same summary are not identical. They are on the same topic, but they may involve different people, places, or decisions.
Check AUC
Summary → text 0.997
Cross-document 0.985

A high score on summary–text pairs is expected, because this is exactly what the model was trained on. The high cross-document score is more interesting, since the model never saw those pairs during training.

3. Semantic textual similarity (STS)

We wrote 100 within-topic sentence pairs on legal topics such as debt, land dispute, and embezzlement. Together they give 200 sentences and 19,900 sentence pairs: the 100 matched pairs are positives and the 19,800 cross-topic pairs are negatives. None of these sentences were in the training data. We report retrieval metrics (each sentence is a query for its partner; MRR with 95% bootstrap CIs over topics) and ROC-AUC. The cross-register AUC compares the 100 positives with the negatives. We also report PR-AUC, because 100 positives against 19,800 negatives can give a high ROC-AUC even when precision is low.

Metric OTA-SBERTurk-Court BERTurk (base) SerBERT
MRR (95% CI) 0.806 [0.740, 0.866] 0.331 [0.276, 0.389] 0.566 [0.502, 0.632]
Recall@1 0.745 0.235 0.465
Cross-register ROC-AUC (95% CI) 0.972 [0.956, 0.985] 0.683 [0.657, 0.713] 0.870 [0.841, 0.900]
PR-AUC 0.521 0.009 0.075

4. Questionnaire-item classification (QIC)

This task checks whether the embeddings keep the psychological constructs apart. There are 47 questionnaire items (honor, individualism, collectivism). Each item and its five aligned proxy sentences count as separate samples, 282 in total (150 honor, 66 individualism, 66 collectivism). We train an RBF-kernel SVM (C = 1, balanced class weights, no tuning) on the embeddings and use 10-fold cross-validation grouped by item, so an item and its proxy sentences never end up in both the training and test folds. Values are the mean ± SD across folds.

Model Accuracy Macro-F1
OTA-SBERTurk-Court 89.58% ± 5.57 0.838
SerBERT 85.25% ± 8.35 0.781
BERTurk (base) 85.08% ± 7.74 0.777

Limitations

  • The model was trained and evaluated only on Latin-transcribed records from Istanbul courts, 1513–1911.
  • Inputs are cut at 256 tokens. For longer documents, split them into sentences first.
  • Larger general-purpose embedding models, such as Qwen3-Embedding 8B, score higher on the STS and questionnaire tasks. OTA-SBERTurk-Court is much smaller and faster than they are, and its gain is over the base model and on retrieval of Ottoman court records.
  • Court records are formulaic and written by an institution. Similarity can therefore reflect the legal genre (for example manumission or endowment deeds) instead of the concept you are looking for.

License

CC-BY-NC-4.0 (non-commercial use). The base model BERTurk is released under the MIT license. The training corpus belongs to İSAM / İstanbul Medipol University and is not redistributed.

Contact

Enes Yılandiloğlu: enes.yilandiloglu@helsinki.fi

Downloads last month
70
Safetensors
Model size
0.1B params
Tensor type
F32
·
Inference Providers NEW
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for enesyila/ota-sbert-kadisicilleri

Finetuned
(219)
this model