ViswanthSai/qwen0.6B-ViswanthSai-lora

Qwen3-0.6B-Python-Code-Assistant

A fine-tuned Qwen3-0.6B model specialized for Python code generation and explanation, trained on the Python Code Instructions dataset.

Model Details

• Developed by: ViswanthSai
• Model type: Language Model with LoRA adapters
• Language(s): English, Python
• License: [Same as base model - Qwen/Qwen3-0.6B]
• Finetuned from model: Qwen/Qwen3-0.6B

Uses

Direct Use

This model is designed to:

• Generate Python code based on natural language instructions
• Explain Python code concepts
• Answer programming questions related to Python
• Assist with debugging and problem-solving in Python

Downstream Use

The model can be integrated into:

• Code editors and IDEs as an AI assistant
• Educational platforms for teaching Python
• Developer tools for code completion and documentation

Out-of-Scope Use

This model should not be used for:

• Production-level code without human review
• Generating code for malicious purposes
• As a replacement for professional programming expertise

How to Get Started with the Model

from transformers import AutoTokenizer, AutoModelForCausalLM
from peft import PeftModel
import torch

# Load base model and tokenizer
model_id = "Qwen/Qwen3-0.6B"
adapter_id = "ViswanthSai/qwen0.6B-ViswanthSai-lora"  # Your adapter repo name

tokenizer = AutoTokenizer.from_pretrained(model_id)

# Load model with adapters
from transformers import BitsAndBytesConfig
bnb_config = BitsAndBytesConfig(load_in_4bit=True)

base_model = AutoModelForCausalLM.from_pretrained(
    model_id, 
    quantization_config=bnb_config,
    device_map="auto",
    trust_remote_code=True
)
model = PeftModel.from_pretrained(base_model, adapter_id)
model.eval()

# Example prompt
prompt = "Instruction: Write a function to calculate the Fibonacci sequence. Output:"
input_ids = tokenizer(prompt, return_tensors="pt").to(model.device)

# Generate
with torch.no_grad():
    output = model.generate(**input_ids, max_new_tokens=256, do_sample=True, temperature=0.7)
    
result = tokenizer.decode(output[0], skip_special_tokens=True)
print(result)

Training Details

Training Data

This model was fine-tuned on the iamtarun/python_code_instructions_18k_alpaca dataset, which contains:

• 18,612 Python code examples
• Instructions and corresponding code outputs
• Various Python programming concepts and tasks

Training Procedure

• Framework: Unsloth + Hugging Face Transformers
• Techniques: LoRA (Low-Rank Adaptation)
• Duration: ~1 hour on Google Colab Pro (T4 GPU)

Training Hyperparameters

• LoRA rank: 16
• LoRA alpha: 32
• LoRA dropout: 0.05
• Target modules: q_proj, k_proj, v_proj, o_proj
• Batch size: 6 per device, 24 effective (with gradient accumulation)
• Sequence length: 128 tokens
• Epochs: 2
• Learning rate: 2e-5
• Optimizer: Paged AdamW 8-bit
• LR scheduler: Cosine
• Mixed precision: FP16
• Weight decay: 0.01

Evaluation

The model was evaluated on a test split (1% of dataset) during training. The training loss converged from ~4.8 to ~2.9 by the end of training.

Environmental Impact

• Hardware Used: Google Colab Pro (Tesla T4 GPU)
• Hours used: ~1 hour
• Cloud Provider: Google
• Carbon Emitted: Minimal due to short training time and small model size

Technical Specifications

Model Architecture

• Base model: Qwen3-0.6B (6 billion parameters)
• LoRA adapters only train 0.08% of parameters (4,587,520 parameters)

Compute Infrastructure

• Google Colab Pro
• Tesla T4 GPU (16GB VRAM)
• 4-bit quantization for memory efficiency

Framework Versions

• PEFT: 0.15.2
• Transformers: 4.51.3
• Unsloth: 2025.5.6
• PyTorch: 2.7.0+cu126