tengfeima-ai/dreambooth-sdxl-tfm-portrait

DreamBooth LoRA — Personal Portrait Subject (SDXL)

This is a LoRA fine-tuned version of Stable Diffusion XL 1.0 using the DreamBooth technique to teach the model a specific personal identity.

• Identifier token: tfm
• Class: person
• Base model: stabilityai/stable-diffusion-xl-base-1.0
• Training technique: DreamBooth + LoRA (rank=32, applied to UNet + text encoders)
• Training steps: 1500
• Training images: 5 selfie portraits (1024×1024)
• Prior preservation: 200 class images

📊 WandB Training Run: https://wandb.ai/tengfeima/dreambooth-lora-sd-xl/runs/5s7v8lrc

---

Usage

import torch
from diffusers import StableDiffusionXLPipeline

# Disable buggy cudnn_sdp backend on newer PyTorch builds
torch.backends.cuda.enable_cudnn_sdp(False)

pipe = StableDiffusionXLPipeline.from_pretrained(
    "stabilityai/stable-diffusion-xl-base-1.0",
    torch_dtype=torch.float16,
)
pipe.load_lora_weights("tengfeima-ai/dreambooth-sdxl-tfm-portrait")
pipe = pipe.to("cuda")

# Generate with the identifier token
image = pipe(
    "a photo of tfm person smiling, natural lighting, high quality portrait",
    num_inference_steps=30,
    guidance_scale=7.5,
).images[0]
image.save("output.png")

⚠️ Note: If you are using PyTorch ≥ 2.11.0, you may need to add torch.backends.cuda.enable_cudnn_sdp(False) before loading the model to avoid a cuDNN SDPA backend error.

---

Prompt Tips

Use tfm person in your prompt to trigger the learned identity:

Prompt	Expected Result
a photo of tfm person, natural lighting	Realistic portrait
a painting of tfm person in the style of Van Gogh	Artistic style transfer
a photo of tfm person hiking in the mountains	New environment
a photo of a person smiling	Generic person (prior preserved)

---

Training Details

Hyperparameter	Value
Base Model	SDXL 1.0
LoRA Rank	32
Learning Rate	1e-4 (constant)
Steps	1500
Batch Size	1 (grad accum 4, effective 4)
Mixed Precision	bf16
Resolution	1024×1024
Text Encoder	Fine-tuned (LoRA)
Prior Loss Weight	1.0
GPU	NVIDIA H100 80GB
Training Time	~82 minutes
Final Loss	0.00828

---

Repository Contents

├── pytorch_lora_weights.safetensors   # LoRA weights (227 MB, rank=32)
├── training_images/                   # 5 instance images used for training
└── samples/
    ├── neutral_portrait/              # 2 images — close to training distribution
    ├── artistic_style/               # 2 images — Van Gogh & cyberpunk styles
    ├── different_environment/        # 2 images — mountains & beach
    └── class_test_prior/             # 2 images — generic person (prior preservation test)

---

License

Apache 2.0 — feel free to use, modify, and redistribute with attribution.