Korean Patent QA - LLaMA-3 QLoRA Adapter

Model Description

This is a QLoRA adapter for LLaMA-3-Korean-Bllossom-8B, fine-tuned on 8,502 Korean patent and trademark decision documents from the Korean Intellectual Property Tribunal. The model specializes in answering questions about Korean patent law, trademark law, and design rights with high accuracy and confidence.

Key Features

🎯 81% Token Accuracy on validation set
📉 Low Entropy (0.665) indicating confident predictions
💾 Memory Efficient using QLoRA (4-bit quantization)
🏛️ Legal Domain Expertise in intellectual property law
🇰🇷 Korean Language optimized for Korean legal terminology

Model Details

Base Model

Model: MLP-KTLim/llama-3-Korean-Bllossom-8B
Architecture: LLaMA-3
Parameters: 8 Billion
Language: Korean

Training

Method: QLoRA (Quantized Low-Rank Adaptation)
Quantization: 4-bit (NF4) with double quantization
LoRA Rank: 16
LoRA Alpha: 32
Target Modules: q_proj, k_proj, v_proj, o_proj, gate_proj, up_proj, down_proj
Training Time: ~7.5 hours (3 epochs)
GPU: NVIDIA GPU with CUDA support

Dataset

Source: Korean Intellectual Property Tribunal Decision Documents
Size: 8,502 QA pairs
Split: 90% train (7,651), 10% validation (851)
Domain: Patent Law, Trademark Law, Design Rights
Format: Question-Answer pairs with legal citations
Language: Korean

Performance

Metric	Value	Description
Token Accuracy	81.0%	Percentage of correctly predicted tokens
Validation Loss	0.651	Cross-entropy loss on validation set
Entropy	0.665	Low entropy indicates confident predictions
Training Loss	0.494	Final training loss (epoch 3)

Training Curve

The model showed consistent improvement across 3 epochs:

Epoch 1: Loss 0.777 → 0.612, Accuracy 78.2% → 80.2%
Epoch 2: Loss 0.589 → 0.480, Accuracy 80.9% → 81.0%
Epoch 3: Loss stabilized at 0.494, Accuracy maintained at 81.0%

Usage

Installation

pip install torch transformers peft bitsandbytes accelerate

Quick Start

import torch
from transformers import AutoTokenizer, AutoModelForCausalLM, BitsAndBytesConfig
from peft import PeftModel

# Configuration for 4-bit quantization
bnb_config = BitsAndBytesConfig(
    load_in_4bit=True,
    bnb_4bit_use_double_quant=True,
    bnb_4bit_quant_type="nf4",
    bnb_4bit_compute_dtype=torch.bfloat16
)

# Load base model
base_model_name = "MLP-KTLim/llama-3-Korean-Bllossom-8B"
tokenizer = AutoTokenizer.from_pretrained(base_model_name)
base_model = AutoModelForCausalLM.from_pretrained(
    base_model_name,
    quantization_config=bnb_config,
    device_map="auto",
    torch_dtype=torch.bfloat16,
)

# Load LoRA adapter
model = PeftModel.from_pretrained(base_model, "tree193nn/patent-qa-llama3-qlora")
model.eval()

# Inference
question = "상표권의 보호기간은 얼마나 되나요?"

prompt = f"""<|begin_of_text|><|start_header_id|>system<|end_header_id|>

당신은 지식재산권법 전문가입니다. 특허 및 상표 관련 법률 질문에 대해 정확하고 상세한 답변을 제공해주세요.<|eot_id|><|start_header_id|>user<|end_header_id|>

{question}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

"""

inputs = tokenizer(prompt, return_tensors="pt").to(model.device)
outputs = model.generate(
    **inputs,
    max_new_tokens=256,
    temperature=0.7,
    top_p=0.9,
    do_sample=True,
    pad_token_id=tokenizer.eos_token_id,
)
answer = tokenizer.decode(outputs[0], skip_special_tokens=True)
print(answer)

Example Questions

questions = [
    "상표권의 보호기간은 얼마나 되나요?",  # How long is trademark protection?
    "특허 출원 시 필요한 서류는 무엇인가요?",  # What documents are needed for patent filing?
    "디자인권과 저작권의 차이는 무엇인가요?",  # What's the difference between design rights and copyright?
]

Intended Use

Primary Use Cases

✅ Question Answering about Korean intellectual property law ✅ Legal Research assistance for patent and trademark matters ✅ Educational Tool for learning Korean IP law ✅ Information Retrieval from patent decision documents

Out-of-Scope Use

❌ Legal Advice: This model should NOT be used as a substitute for professional legal counsel ❌ Official Decisions: Outputs are not legally binding ❌ Non-Korean Languages: Optimized for Korean only ❌ General Purpose QA: Specializes in IP law, may not perform well on general topics

Limitations

Known Limitations

Temporal Limitation: Training data is from before 2024; recent legal changes may not be reflected
Domain Specificity: Performs best on patent/trademark law; limited on other legal areas
Language: Optimized for Korean; English or other languages not supported
Hallucination Risk: May generate plausible but incorrect legal interpretations
Context Length: Limited to 2048 tokens due to training configuration

Bias and Fairness

Training Data Bias: Reflects biases present in Korean Intellectual Property Tribunal decisions
Geographic Focus: Specific to Korean law; not applicable to other jurisdictions
Language Bias: Korean legal terminology heavily featured

Ethical Considerations

⚠️ Important Notice: This model is intended for research and educational purposes only. Users should:

Verify Information: Always cross-reference with official legal sources
Seek Professional Advice: Consult qualified legal professionals for actual cases
Understand Limitations: Recognize the model's domain and temporal constraints
Use Responsibly: Do not use for misleading or fraudulent purposes

Technical Specifications

Hardware Requirements

Minimum:

GPU: 16GB VRAM (e.g., NVIDIA RTX 4080, A4000)
RAM: 32GB
Storage: 20GB (base model + adapter)

Recommended:

GPU: 24GB+ VRAM (e.g., NVIDIA RTX 4090, A5000, A6000)
RAM: 64GB
Storage: 30GB

Software Requirements

Python 3.8+
PyTorch 2.0+
Transformers 4.38.0+
PEFT 0.8.0+
BitsAndBytes 0.42.0+
CUDA 11.8+ or 12.0+

Training Hyperparameters

# QLoRA Configuration
lora_r: 16
lora_alpha: 32
lora_dropout: 0.05
quantization: 4-bit (NF4)

# Training Configuration
epochs: 3
batch_size: 2 (per device)
gradient_accumulation_steps: 8
effective_batch_size: 16
learning_rate: 2e-4
lr_scheduler: cosine
warmup_ratio: 0.03
weight_decay: 0.01
optimizer: paged_adamw_32bit
max_seq_length: 2048

Evaluation

Metrics Explanation

Token Accuracy (81%): Measures how many individual tokens (words/subwords) match the ground truth
Validation Loss (0.651): Lower is better; indicates model's prediction confidence
Entropy (0.665): Low entropy means the model makes confident predictions rather than being uncertain

Comparison to Base Model

The fine-tuned adapter shows significant improvements in the legal domain:

Better understanding of Korean legal terminology
More accurate citations of relevant laws and regulations
Reduced hallucination on IP law topics

Citation

If you use this model in your research, please cite:

@misc{patent-qa-llama3-qlora,
  author = {tree193nn},
  title = {Korean Patent QA with LLaMA-3 QLoRA},
  year = {2024},
  publisher = {Hugging Face},
  howpublished = {\url{https://huggingface.co/tree193nn/patent-qa-llama3-qlora}},
  note = {QLoRA adapter for Korean patent and trademark law question answering}
}