---
license: mit
language:
- en
base_model:
- TinyLlama/TinyLlama-1.1B-Chat-v1.0
tags:
- tinylama
- fine-tuning
- lora
- qlora
- rag
- evaluation
- research
- pytorch
---

# FT-Lab: TinyLlama-1.1B Fine-Tuning Baselines (Full-FT / LoRA / QLoRA)

A minimal, fully reproducible fine-tuning stack for **TinyLlama-1.1B**,  
providing clean baselines for **Full Fine-Tuning, LoRA, and QLoRA**.  
Designed for **Colab / T4 / A10** environments.

---

# Quick Start

```python
from transformers import AutoTokenizer, AutoModelForCausalLM

model_id = "Sai1202HF/ft-lab-tinyllama-1.1b-finetuning"

tokenizer = AutoTokenizer.from_pretrained(model_id)
model = AutoModelForCausalLM.from_pretrained(model_id, device_map="auto")

inputs = tokenizer("Explain LoRA.", return_tensors="pt")
outputs = model.generate(**inputs, max_new_tokens=128)

print(tokenizer.decode(outputs[0], skip_special_tokens=True))
```

## Model Summary

This model is part of **FT-Lab**, a reproducible research toolkit for TinyLlama fine-tuning.  
It serves as a **baseline reference checkpoint** for comparing Full-FT, LoRA, and QLoRA.

Although FT-Lab includes **RAG and retrieval evaluation pipelines**,  
this uploaded checkpoint is **not** a RAG-optimized model.  
It is intended for instruction tuning and fine-tuning studies.

This model is a fine-tuned variant of **TinyLlama/TinyLlama-1.1B-Chat-v1.0**,  
optimized for small-GPU environments using:

- **Full Fine-Tuning**  
- **LoRA**  
- **QLoRA**

The goal is to provide a **minimal, reproducible baseline** for:

- instruction tuning  
- RAG evaluation experiments  
- small-scale research  
- controlled ablation studies  
- comparison of FT / LoRA / QLoRA behaviors  

All training and evaluation scripts belong to **FT-LLab**,  
designed for lightweight environments such as **T4** or **A10 GPUs**.

# Intended Use

This model is intended for:

- Educational understanding of fine-tuning pipelines
- Baseline research experiments
- Small-scale RAG + instruction tuning studies
- Method comparison of Full FT / LoRA / QLoRA

Not intended for high-risk domains (finance, healthcare, legal) without further evaluation and safeguards.

# Training Procedure

- **Base Model:** TinyLlama/TinyLlama-1.1B-Chat-v1.0  
- **Training Methods:** Full Fine-Tuning / LoRA / QLoRA  
- **Hardware:** T4 / A10 / Colab Pro  
- **Framework:** PyTorch + HuggingFace Transformers  
- **Optimizer:** AdamW  

Hyperparameters and scripts are fully reproducible inside FT-Lab.

# Dataset

All datasets in the public demo (`toy_qa.jsonl`, `sample_eval.jsonl`) are **synthetic dummy datasets** used solely to demonstrate the FT-Lab pipeline.

They **do not represent meaningful real-world semantic content**.

For real experiments, replace the dataset under `data/` with your own or a public instruction-tuning dataset.

# Evaluation

FT-Lab includes evaluation using:

- Exact Match (EM)
- Token-level accuracy
- BERTScore (optional)
- Custom RAG verification pipeline

These metrics support **relative comparison** between methods, not benchmark-grade scores.

# Limitations

- Model size is small (1.1B), limiting reasoning and factual accuracy.
- Training data in the demo is synthetic.
- May hallucinate or produce incorrect content.
- Safety alignment is minimal.

# Ethical Considerations

Use in safety-critical or high-risk settings requires:

- Additional evaluation
- Guardrails / filtering
- Human oversight

# Citation

If you use FT-Lab or this model, please cite the repository:

**https://github.com/REICHIYAN/ft_lab**