|
|
--- |
|
|
license: mit |
|
|
datasets: |
|
|
- nyu-mll/multi_nli |
|
|
- stanfordnlp/snli |
|
|
language: |
|
|
- en |
|
|
metrics: |
|
|
- accuracy |
|
|
base_model: |
|
|
- answerdotai/ModernBERT-large |
|
|
- tasksource/ModernBERT-large-nli |
|
|
pipeline_tag: text-classification |
|
|
library_name: sentence-transformers |
|
|
tags: |
|
|
- cross-encoder |
|
|
- modernbert |
|
|
- mnli |
|
|
- snli |
|
|
--- |
|
|
# ModernBERT Cross-Encoder: Natural Language Inference (NLI) |
|
|
|
|
|
This cross encoder performs sequence classification for contradiction/neutral/entailment labels. This has |
|
|
drop-in compatibility with comparable sentence transformers cross encoders. |
|
|
|
|
|
I trained this model by initializaing the ModernBERT-large weights from the brilliant `tasksource/ModernBERT-large-nli` |
|
|
zero-shot classification model. Then I trained it with a batch size of 64 using the `sentence-transformers` AllNLI |
|
|
dataset. |
|
|
|
|
|
For the `large` version, I froze all layers initialized from the tasksource model up to 19, and fine tuned only the |
|
|
remaining layers with a new classification head. |
|
|
|
|
|
--- |
|
|
|
|
|
## Features |
|
|
- **High performing:** Achieves **92.02%** and 91.10% on MNLI mismatched and SNLI test. |
|
|
- **Efficient architecture:** Based on the ModernBERT-large design (395M parameters), offering faster inference speeds. |
|
|
- **Extended context length:** Processes sequences up to 8192 tokens, great for LLM output evals. |
|
|
|
|
|
--- |
|
|
|
|
|
# NLI Evaluation Results |
|
|
|
|
|
F1-Micro scores (equivalent to accuracy) for each dataset. |
|
|
Performance was measured at bs=32 using a Nvidia Blackwell PRO 6000 Max-Q. |
|
|
|
|
|
| Model | finecat | mnli | mnli_mismatched | snli | anli_r1 | anli_r2 | anli_r3 | wanli | lingnli | Throughput (samples/s) | Peak GPU Mem (MB) | |
|
|
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | |
|
|
| `MoritzLaurer/DeBERTa-v3-large-mnli-fever-anli-ling-wanli` | **0.8233** | <u>0.9121</u> | 0.9079 | 0.8898 | **0.7960** | **0.6830** | **0.6400** | <u>0.7700</u> | **0.8821** | 454.96 | 3250.44 | |
|
|
| `dleemiller/finecat-nli-l` | <u>0.8227</u> | **0.9152** | **0.9265** | 0.9162 | <u>0.7480</u> | <u>0.5700</u> | <u>0.5433</u> | **0.7706** | <u>0.8742</u> | 539.04 | 1838.06 | |
|
|
| `tasksource/ModernBERT-large-nli` | 0.7959 | 0.8983 | <u>0.9229</u> | 0.9188 | 0.7260 | 0.5110 | 0.4925 | 0.6978 | 0.8504 | 543.44 | 1838.06 | |
|
|
| `dleemiller/ModernCE-large-nli` | 0.7811 | 0.9088 | 0.9205 | **0.9273** | 0.6630 | 0.4860 | 0.4408 | 0.6576 | 0.8566 | 540.74 | 1838.06 | |
|
|
| `cross-encoder/nli-deberta-v3-large` | 0.7618 | 0.9019 | 0.9049 | <u>0.9220</u> | 0.5300 | 0.4170 | 0.3758 | 0.6548 | 0.8466 | 448.35 | 3250.44 | |
|
|
|
|
|
|
|
|
--- |
|
|
|
|
|
## Usage |
|
|
|
|
|
To use ModernCE for NLI tasks, you can load the model with the Hugging Face `sentence-transformers` library: |
|
|
|
|
|
```python |
|
|
from sentence_transformers import CrossEncoder |
|
|
|
|
|
# Load ModernCE model |
|
|
model = CrossEncoder("dleemiller/ModernCE-large-nli") |
|
|
|
|
|
scores = model.predict([ |
|
|
('A man is eating pizza', 'A man eats something'), |
|
|
('A black race car starts up in front of a crowd of people.', 'A man is driving down a lonely road.') |
|
|
]) |
|
|
|
|
|
# Convert scores to labels |
|
|
label_mapping = ['contradiction', 'entailment', 'neutral'] |
|
|
labels = [label_mapping[score_max] for score_max in scores.argmax(axis=1)] |
|
|
# ['entailment', 'contradiction'] |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## Training Details |
|
|
|
|
|
### Pretraining |
|
|
We initialize the `tasksource/ModernBERT-large` weights. |
|
|
|
|
|
Details: |
|
|
- Batch size: 64 |
|
|
- Learning rate: 3e-4 |
|
|
- **Attention Dropout:** attention dropout 0.1 |
|
|
|
|
|
### Fine-Tuning |
|
|
Fine-tuning was performed on the SBERT AllNLI.tsv.gz dataset. |
|
|
|
|
|
### Validation Results |
|
|
The model achieved the following test set performance after fine-tuning: |
|
|
- **MNLI Unmatched:** 0.9202 |
|
|
- **SNLI:** 0.9110 |
|
|
|
|
|
--- |
|
|
|
|
|
## Model Card |
|
|
|
|
|
- **Architecture:** ModernBERT-large |
|
|
- **Fine-Tuning Data:** `sentence-transformers` - AllNLI.tsv.gz |
|
|
|
|
|
--- |
|
|
|
|
|
## Thank You |
|
|
|
|
|
Thanks to the AnswerAI team for providing the ModernBERT models, and the Sentence Transformers team for their leadership in transformer encoder models. |
|
|
We also thank the tasksource team for their work on zeroshot encoder models. |
|
|
|
|
|
--- |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you use this model in your research, please cite: |
|
|
|
|
|
```bibtex |
|
|
@misc{moderncenli2025, |
|
|
author = {Miller, D. Lee}, |
|
|
title = {ModernCE NLI: An NLI cross encoder model}, |
|
|
year = {2025}, |
|
|
publisher = {Hugging Face Hub}, |
|
|
url = {https://huggingface.co/dleemiller/ModernCE-large-nli}, |
|
|
} |
|
|
``` |
|
|
|
|
|
--- |
|
|
|
|
|
## License |
|
|
|
|
|
This model is licensed under the [MIT License](LICENSE). |
