denserepr-code-48M-pretrain-EN-v1

A small dense embedding model for code retrieval: 48M parameters, one 384-d vector per text, cosine similarity, 8,192-token context. It scores 64.84 mean nDCG@10 on CoIR (10 tasks), +11.4 over the model it was trained from.

It was made by continued contrastive pretraining of ibm-granite/granite-embedding-small-english-r2 on ~23M code retrieval pairs with in-batch negatives only. This is the pretraining stage: no hard negatives, no distillation and no fine-tuning stage have been applied yet.

Usage

from sentence_transformers import SentenceTransformer

model = SentenceTransformer("f4roukb/denserepr-code-48M-pretrain-EN-v1")

queries = ["read a csv file into a list of dicts"]
docs = [
    "def load(path):\n    import csv\n    with open(path) as f:\n        return list(csv.DictReader(f))",
    "def save(rows, path):\n    import json\n    with open(path, 'w') as f:\n        json.dump(rows, f)",
]
q = model.encode(queries)
d = model.encode(docs)
print(model.similarity(q, d))  # the first document scores higher

No prompts or instructions: queries and documents are encoded the same way.

parameters 47,662,464
embedding 384-d, CLS pooling, cosine similarity
max sequence length 8,192 tokens
architecture ModernBERT (from granite-embedding-small-english-r2)
weights fp32 safetensors
precision mixed bf16 / fp32: trained with fp32 weights and bf16 autocast; CoIR scores measured in bf16

Results: CoIR (MTEB)

nDCG@10 Γ—100, measured at 8,192 tokens with mteb 2.27.8 task revisions. The base model is measured here the same way.

task base this model
AppsRetrieval 13.48 30.75
CodeFeedbackMT 52.20 81.19
CodeFeedbackST 76.84 81.26
CodeSearchNetCCRetrieval 48.42 75.97
CodeTransOceanContest 77.63 80.00
CodeTransOceanDL 33.63 34.84
CosQA 35.40 33.84
COIRCodeSearchNetRetrieval 60.42 79.90
StackOverflowQA 90.04 90.97
SyntheticText2SQL 46.44 59.69
mean 53.45 64.84

Among the dense (single-vector) models on the MTEB CoIR leaderboard with all 10 tasks, it is the best one under 80M parameters (as of 2026-11-12).

Read these scores with the following in mind:

  • Not zero-shot on CoIR. The training data includes the CoIR train splits. The training set was decontaminated against the CoIR test sets (exact match after normalization, in both roles).

Data

source content pairs before cleaning
lightonai/cornstack docstring β†’ code: Python, Go, Java, PHP, JavaScript, Ruby 21.5M
codefuse-ai/F2LLM-v2 OCGI (instruction β†’ Python) and xCodeEval (problem β†’ solution) 1.1M
CoIR-Retrieval train splits the 10 CoIR tasks 2.2M

Cleaning: exact deduplication of normalized pairs, removal of stub code (pass, empty bodies, return true) with tree-sitter in 14 languages, and decontamination against the CoIR test sets and NanoCoIR. 24.8M pairs β†’ 23.1M.

Limitations

  • English queries; code mostly in the six cornstack languages plus SQL.
  • Weakest relative to peers on short natural-language queries (CosQA is below the base model).
  • A pretraining-stage model: a fine-tuning stage with hard negatives would likely change the scores.

License and attribution

Apache-2.0. This model is a modified version of ibm-granite/granite-embedding-small-english-r2 (Apache-2.0, Β© IBM): its weights were changed by the training described above. It is not an IBM model.

Training data and licenses: cornstack (Apache-2.0), F2LLM-v2 (Apache-2.0; its OCGI and xCodeEval parts are CC-BY-4.0), and the CoIR train splits, whose upstream sources are APPS (MIT), CodeSearchNet, Code-Feedback (Apache-2.0), gretelai synthetic text-to-SQL (Apache-2.0), CodeTransOcean, CoSQA and StackOverflow (CC-BY-SA). Thanks to their authors.

Citation

@misc{denserepr2026,
  title  = {denserepr-code-48M-pretrain-EN-v1: a small dense embedding model for code retrieval},
  author = {Farouk B.},
  year   = {2026},
  url    = {https://huggingface.co/f4roukb/denserepr-code-48M-pretrain-EN-v1}
}
Downloads last month
59
Safetensors
Model size
47.7M params
Tensor type
F32
Β·
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support

Model tree for f4roukb/denserepr-code-48M-pretrain-EN-v1

Finetuned
(3)
this model

Datasets used to train f4roukb/denserepr-code-48M-pretrain-EN-v1

Space using f4roukb/denserepr-code-48M-pretrain-EN-v1 1