|
|
--- |
|
|
license: mit |
|
|
base_model: |
|
|
- thomas-sounack/BioClinical-ModernBERT-base |
|
|
tags: |
|
|
- sentence-transformers |
|
|
- sentence-similarity |
|
|
- medical |
|
|
- clinical |
|
|
- biomedical |
|
|
- pubmed |
|
|
- healthcare |
|
|
- medical-ai |
|
|
- clinical-nlp |
|
|
- bioinformatics |
|
|
- medical-literature |
|
|
- clinical-text |
|
|
--- |
|
|
# Clinical ModernBERT Embedding Model |
|
|
|
|
|
A specialized medical embedding model fine-tuned from Clinical ModernBERT using InfoNCE contrastive learning on PubMed title-abstract pairs. |
|
|
|
|
|
## Model Details |
|
|
|
|
|
- **Base Model**: thomas-sounack/BioClinical-ModernBERT-base |
|
|
- **Training Method**: InfoNCE contrastive learning |
|
|
- **Training Data**: PubMed title-abstract pairs |
|
|
- **Max Sequence Length**: 2048 tokens |
|
|
|
|
|
## Usage |
|
|
|
|
|
```python |
|
|
from sentence_transformers import SentenceTransformer |
|
|
|
|
|
# Load the model |
|
|
model = SentenceTransformer("lokeshch19/ModernPubMedBERT") |
|
|
|
|
|
# Encode medical texts |
|
|
texts = [ |
|
|
"Rheumatoid arthritis is an autoimmune disorder attacking joint linings.", |
|
|
"Inflammatory cytokines in RA lead to progressive cartilage and bone destruction." |
|
|
] |
|
|
embeddings = model.encode(texts) |
|
|
``` |
|
|
|
|
|
## Applications |
|
|
|
|
|
- Medical document similarity analysis |
|
|
- Clinical text retrieval systems |
|
|
- Biomedical literature search |
|
|
- Medical concept matching and classification |
|
|
|
|
|
## Model Comparison |
|
|
|
|
|
Compared to `NeuML/bioclinical-modernbert-base-embeddings`, our model demonstrates superior understanding of medical concepts and enhanced discrimination of non-medical content. |
|
|
|
|
|
### Comprehensive Evaluation Results |
|
|
|
|
|
| Metric | Our Model | NeuML Model | Improvement | |
|
|
|--------|-----------|-------------|-------------| |
|
|
| **Accuracy@1** | 91.28% | 85.86% | +6.3% | |
|
|
| **Accuracy@3** | 98.46% | 95.66% | +2.9% | |
|
|
| **Accuracy@5** | 99.24% | 97.14% | +2.2% | |
|
|
| **Accuracy@10** | 99.64% | 98.29% | +1.4% | |
|
|
| **NDCG@5** | 95.96% | 92.37% | +3.9% | |
|
|
| **NDCG@10** | 96.10% | 92.75% | +3.6% | |
|
|
| **MRR@10** | 94.89% | 90.90% | +4.4% | |
|
|
| **MAP@100** | 94.91% | 90.96% | +4.3% | |
|
|
|
|
|
*Evaluation performed using `InformationRetrievalEvaluator` from sentence-transformers on the `gamino/wiki_medical_terms` dataset.* |
|
|
|
|
|
## Model Comparison |
|
|
|
|
|
Compared to `NeuML/bioclinical-modernbert-base-embeddings`, this model demonstrates superior understanding of medical concepts and enhanced discrimination of non-medical content. |
|
|
|
|
|
### Medical Text Similarity |
|
|
|
|
|
**Example 1: Related Medical Concepts** |
|
|
```python |
|
|
text1 = "Hypertension increases the risk of stroke and heart attack." |
|
|
text2 = "High blood pressure damages arterial walls over time, leading to cardiovascular events." |
|
|
|
|
|
# Cosine Similarity Results: |
|
|
# Our Model: 0.5941 (59.4%) |
|
|
# NeuML Model: 0.5267 (52.7%) |
|
|
# Improvement: +12.7% |
|
|
``` |
|
|
|
|
|
### Non-Medical Text Discrimination |
|
|
|
|
|
**Example 2: Medical vs. Programming Terms** |
|
|
```python |
|
|
texts = ["diabetes type 2", "asyncio.run()"] |
|
|
|
|
|
# Cosine Similarity Results: |
|
|
# Our Model: 0.0804 (8.0%) - Correctly identifies low similarity |
|
|
# NeuML Model: 0.1926 (19.3%) - Higher false similarity |
|
|
# Better Discrimination: 58% lower false positive rate |
|
|
``` |
|
|
|
|
|
### Key Advantages |
|
|
|
|
|
- **Enhanced Medical Understanding**: 12.7% better similarity detection for related medical concepts |
|
|
- **Improved Discrimination**: 58% reduction in false similarities between medical and non-medical terms |
|
|
- **Domain Specialization**: Fine-tuned specifically on PubMed literature for optimal medical text processing |
|
|
|
|
|
## Training Details |
|
|
|
|
|
- **Optimizer**: AdamW (learning rate: 3e-4, weight decay: 0.1) |
|
|
- **Batch Size**: 72 |
|
|
- **Training Steps**: 7,000 |
|
|
- **Warmup Steps**: 700 |
|
|
|
|
|
## Citation |
|
|
|
|
|
If you use this model, please cite the base model paper and acknowledge this fine-tuning work. |
