tahp0604/finbert-sentfin-lora

FinBERT-LoRA-SentFin

This model is a Parameter-Efficient Fine-Tuning (PEFT) version of FinBERT specifically optimized for high-precision financial sentiment analysis. It was fine-tuned using Low-Rank Adaptation (LoRA) on the SentFin 1.0 dataset.

Model Details

Model Description

The FinBERT-LoRA-SentFin adapter adds a specialized sentiment-aware layer to the base FinBERT model. By using LoRA, the model retains the deep financial domain knowledge of the original ProsusAI/finbert while adapting its attention mechanisms to the specific nuances of the SentFin 1.0 corpus.

• Developed by: tahp0604
• Model type: PEFT / LoRA (Low-Rank Adaptation)
• Base Model: ProsusAI/finbert
• Language(s): English
• License: MIT
• Finetuned from model: ProsusAI/finbert
• Dataset: SentFin 1.0

Loading and Usage

To use this adapter, you must load the base model first and then attach the adapter.

from transformers import AutoModelForSequenceClassification, AutoTokenizer
from peft import PeftModel
import torch

# 1. Define IDs
base_model_id = "ProsusAI/finbert"
adapter_model_id = "tahp0604/finbert-sentfin-lora" # or local path

# 2. Load Tokenizer
tokenizer = AutoTokenizer.from_pretrained(base_model_id)

# 3. Load Base Model
base_model = AutoModelForSequenceClassification.from_pretrained(base_model_id)

# 4. Load and Attach PEFT Adapter
model = PeftModel.from_pretrained(base_model, adapter_model_id)

# Example Inference
text = "Reliance fell 2.1% today amid refinery outage concerns."
inputs = tokenizer(text, return_tensors="pt")

model.eval()
with torch.no_grad():
    logits = model(**inputs).logits
    predicted_class_id = logits.argmax().item()

# Map to label
labels = {0: "positive", 1: "negative", 2: "neutral"}
print(f"Sentiment: {labels[predicted_class_id]}")

Training Details

Training Data

The model was trained on the SentFin 1.0 dataset, which consists of over 10,000 financial news headlines annotated by domain experts for entity-level sentiment.

Training Procedure

PEFT / LoRA Configuration

• LoRA Rank (r): 8
• LoRA Alpha ($\alpha$): 16
• Target Modules: query, value (Attention layers)
• LoRA Dropout: 0.1
• Task Type: Sequence Classification (SEQ_CLS)
• Modules to Save: classifier, score

Training Hyperparameters

• Training regime: FP16 Mixed Precision
• Optimizer: AdamW
• Batch Size: 32 (suggested)
• Learning Rate: 2e-4 (LoRA recommended)

Evaluation

Testing Data

Evaluated on a 20% hold-out split of the SentFin 1.0 dataset.

Metrics

• Accuracy: Primary metric for classification.
• F1 Score: Weighted F1 score to account for class distribution.

Technical Specifications

Model Architecture

• Backbone: BERT-base-uncased (FinBERT)
• PEFT Method: LoRA
• Learnable Parameters: ~0.3M (approx. 0.27% of base model)
• Total Parameters: ~110M

Model Card Contact

tahp0604

Framework versions

• PEFT 0.18.1
• Transformers 4.x
• PyTorch 2.x