Mitchins/deberta-v3-small-literary-explicitness

Literary Content Classifier - DeBERTa v3 Small (v2.0)

An improved fine-tuned DeBERTa-v3-small model for sophisticated literary content analysis across 7 categories of explicitness. This v2.0 model features significant improvements over the original, including focal loss training, extended epochs, and data quality enhancements.

🚀 Key Improvements in v2.0

• +4.5% accuracy improvement (81.8% vs 77.3%)
• +6.4% macro F1 improvement (0.754 vs 0.709)
• +21% improvement on violent content (F1: 0.581 vs 0.478)
• +19% improvement on suggestive content (F1: 0.476 vs 0.400)
• Focal loss training for better minority class performance
• Clean dataset with cross-split contamination resolved
• Extended training (4.79 epochs vs 1.1 epochs)

Model Description

This model provides nuanced classification of textual content across 7 categories, enabling sophisticated analysis for digital humanities, content curation, and literary research applications.

Categories

ID	Category	Description	F1 Score
0	EXPLICIT-DISCLAIMER	Content warnings and age restriction notices	0.977
1	EXPLICIT-OFFENSIVE	Profanity, crude language, offensive content	0.813
2	EXPLICIT-SEXUAL	Graphic sexual content and detailed intimate scenes	0.930
3	EXPLICIT-VIOLENT	Violent or disturbing content	0.581
4	NON-EXPLICIT	Clean, family-friendly content	0.851
5	SEXUAL-REFERENCE	Mentions of sexual topics without graphic description	0.652
6	SUGGESTIVE	Mild innuendo or romantic themes without explicit detail	0.476

Performance Metrics

Overall Performance

• Accuracy: 81.8%
• Macro F1: 0.754
• Weighted F1: 0.816

Detailed Results (Test Set)

                     precision    recall  f1-score   support
EXPLICIT-DISCLAIMER     0.95      1.00      0.98        19
EXPLICIT-OFFENSIVE      0.82      0.88      0.81       414
EXPLICIT-SEXUAL         0.93      0.91      0.93       514
EXPLICIT-VIOLENT        0.44      0.62      0.58        24
NON-EXPLICIT            0.77      0.87      0.85       683
SEXUAL-REFERENCE        0.63      0.73      0.65       212
SUGGESTIVE              0.37      0.46      0.48       134

            accuracy                        0.82      2000
           macro avg    0.65      0.78      0.75      2000
        weighted avg    0.75      0.82      0.82      2000

Training Details

Model Architecture

• Base Model: microsoft/deberta-v3-small
• Parameters: 141.9M (6 layers, 768 hidden, 12 attention heads)
• Vocabulary: 128,100 tokens
• Max Sequence Length: 512 tokens

Training Configuration

• Training Method: Focal Loss (γ=2.0) for class imbalance
• Epochs: 4.79 (early stopped)
• Learning Rate: 5e-5 with cosine schedule
• Batch Size: 16 (effective 32 with gradient accumulation)
• Warmup Steps: 1,000
• Weight Decay: 0.01
• Early Stopping: Patience 5 on macro F1

Dataset

• Total Samples: 119,023 (after deduplication)
• Training: 83,316 samples
• Validation: 17,853 samples
• Test: 17,854 samples
• Data Quality: Cross-split contamination eliminated (2,127 duplicates removed)

Training Environment

• Framework: PyTorch + Transformers
• Hardware: Apple Silicon (MPS)
• Training Time: ~13.7 hours

Usage

from transformers import AutoTokenizer, AutoModelForSequenceClassification, pipeline

# Load model and tokenizer
model_id = "your-username/deberta-v3-small-explicit-classifier-v2"
model = AutoModelForSequenceClassification.from_pretrained(model_id)
tokenizer = AutoTokenizer.from_pretrained(model_id)

# Create classification pipeline
classifier = pipeline(
    "text-classification", 
    model=model, 
    tokenizer=tokenizer,
    return_all_scores=True,
    truncation=True
)

# Single classification
text = "His hand lingered on hers as he helped her from the carriage."
result = classifier(text)
print(f"Top prediction: {result[0]['label']} ({result[0]['score']:.3f})")

# All class probabilities
for class_result in result:
    print(f"{class_result['label']}: {class_result['score']:.3f}")

Recommended Thresholds (F1-Optimized)

For applications requiring specific precision/recall trade-offs:

Class	Optimal Threshold	Precision	Recall	F1
EXPLICIT-DISCLAIMER	0.995	0.950	1.000	0.974
EXPLICIT-OFFENSIVE	0.626	0.819	0.829	0.824
EXPLICIT-SEXUAL	0.456	0.927	0.911	0.919
EXPLICIT-VIOLENT	0.105	0.441	0.625	0.517
NON-EXPLICIT	0.103	0.768	0.874	0.818
SEXUAL-REFERENCE	0.355	0.629	0.726	0.674
SUGGESTIVE	0.530	0.370	0.455	0.408

Model Files

• model.safetensors: Model weights in SafeTensors format
• config.json: Model configuration with proper label mappings
• tokenizer.json, spm.model: SentencePiece tokenizer files
• label_mapping.json: Label ID to name mapping reference

Limitations & Considerations

1. Challenging Distinctions: SUGGESTIVE vs SEXUAL-REFERENCE categories remain difficult to distinguish due to conceptual overlap
2. Minority Classes: EXPLICIT-VIOLENT and SUGGESTIVE classes have lower F1 scores due to limited training data
3. Context Dependency: Short text snippets may lack sufficient context for accurate classification
4. Domain Specificity: Optimized for literary and review content; performance may vary on other text types
5. Language: English text only

Evaluation Artifacts

The model includes comprehensive evaluation materials:

• Confusion matrix visualization
• Per-class precision-recall curves
• ROC curves for all categories
• Calibration analysis
• Recommended decision thresholds

Ethical Use

This model is designed for:

• Academic research and digital humanities
• Content curation and library science applications
• Literary analysis and publishing workflows
• Educational content assessment

Important: This model should be used responsibly with human oversight for content moderation decisions.

Citation

@misc{literary-explicit-classifier-v2-2025,
  title={Literary Content Analysis: Improved Multi-Class Classification with Focal Loss},
  author={Explicit Content Research Team},
  year={2025},
  note={DeBERTa-v3-small fine-tuned for literary explicitness detection}
}

License

This model is released under the Apache 2.0 license.